Configure Alerting#

Alerting allows you to send ClusterWareAI ™ event data to external target systems, such as email, Slack, ServiceNow, and so on. Configuration is required to set up alerts and mutators for your selected targets.

Tip

While you can enable alerting at any time after installing the ClusterWareAI software, it is recommended that you wait until your cluster is in a steady state after install or upgrade.

Prerequisites: Configure Alerting#

Ensure you have the appropriate permissions to send or post alerts to your selected target platforms. Consult the documentation for your target platforms for other requirements.

Configure Alert#

Select the MQTT topic you want to alert on. Use your MQTT client to see what event messages are available. See MQTT Messaging for details. Contact Penguin Solutions if you need additional information about alerting on specific events you are interested in.
Decide where you want to send the alert. See Supported Alert Targets for details or run the following command to see a complete list of supported targets:
```
cw-alertctl schema
```
Use the cw-alertctl tool to determine the required fields for your selected target:
```
cw-alertctl schema <target>
```
The required fields for the specified target are shown. Add a --full argument to see both required and optional fields.

Tip

The schema for all alert output targets lists message as a required field. For alerts, the message comes from either MQTT or a mutator, which is specified using the sources field when creating the alert.
Create an alert:
```
cw-alertctl create name=<alert name> sources=<mqtt or mutator> output=<target> \
   output_config=<fields>
```
Where:
- name is the name of the new alert
- sources is either mqtt:<topic> or mutator:<name>
- output is the target that will receive the alert
- output_config is the list of fields required by the specified output target
Additional optional fields are available. See cw-alertctl for details.
Test the alert:
```
cw-alertctl -i <name> test --message "<test message>" --send
```
Where:
- --message is test message text
- --send delivers the message through the alert output instead of printing the message via the command
The alert sends the test message to the output target.

Configure Mutator (Optional)#

If you are not sure if you need a mutator, create the alert with the MQTT topic as the source and wait for the alert to send with real event data. If you are satisfied with the default MQTT event message, you do not need a mutator. If you want to modify the MQTT event message, create a mutator and update your alert.

Create a mutator:
```
cw-mutatorctl create --content '{
  "name": "<name>",
  "sources": ["<mqtt or mutator>"],
  "format": "{\"message\": \"<message text, with {placeholders}>\"}"
}'
```
Where:
- name is the name of the new mutator
- sources is either mqtt:<topic> or mutator:<name>
- format is valid JSON after any placeholders are replaced
Additional optional fields are available. See cw-mutatorctl for details.
Test the mutator:
```
cw-mutatorctl -i <name> test
```
A JSON object containing the mutator name and sample values for when, topic, and message are shown.
Update the existing alert to use the new mutator instead of the MQTT message:
```
cw-alertctl -i <alert name> update sources=mutator:<mutator name>
```
The alert message now uses the format specified by the mutator instead of the raw MQTT event data.

Example: Alert Using Generic Webhook#

The following simple example configures an alert for Jira using a generic webhook and the Jira API. The alert creates a Bug in Jira when a compute node moves to the ARS Work Queue state.

Review the Jira API documentation for authentication and body formatting requirements.

Review the full schema for a webhook to identify the required and optional fields:

 [admin@head1 ~]$ cw-alertctl schema webhook --full
 {
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "title": "Destination URL"
    },
    "method": {
      "type": "string",
      "enum": [
        "POST",
        "PUT"
      ],
      "title": "HTTP Method"
    },
    "auth_scheme": {
      "type": "string",
      "title": "Authorization header scheme, i.e 'Bearer', 'Basic'. Defaults \
         to 'Bearer'. For 'Basic', auth_token must already be base64(user:token)."
    },
    "auth_token": {
      "type": "string",
      "title": "Sent as 'Authorization: <auth-scheme> <auth_token>'"
    },
    "auth_user": {
      "type": "string",
      "title": "Username for HTTP Basic auth (use with auth_password instead \
         of auth_token)"
    },
    "auth_password": {
      "type": "string",
      "title": "Password/API token for HTTP basic auth"
    },
    "body": {
      "type": "string",
      "title": "Request body template"
    },
    "message": {
      "type": "string",
      "title": "Alert message"
    }
  },
  "dependencies": {
    "auth_user": [
      "auth_password"
    ],
    "auth_password": [
      "auth_user"
    ]
  },
  "error_dependencies": "auth_user and auth_password must be set together.",
  "required": [
    "url",
    "method",
    "message"
  ],
  "additionalProperties": false
}

The bottom of the schema output shows the required fields, which include url, method, and message in this example. The message field in an alert comes from either MQTT or a mutator, which is specified using the sources field when creating an alert.

Create an alert with all MQTT topics related to compute nodes entering the ARS Work Queue state as the sources and Jira as the output target via webhook:

cw-alertctl create --content '{
   "name": "jira_bug_node_work_queue",
   "description": "Create bug in Jira when compute nodes move to work queue state.",
   "sources": [
       "mqtt:ars/v1/+/events/provisioning_error",
       "mqtt:ars/v1/+/events/drain_failure",
       "mqtt:ars/v1/+/events/require_human",
       "mqtt:ars/v1/+/events/require_manual_fix"
   ],
   "output": "webhook",
   "output_config": {
     "url": "https://mycompany.atlassian.net/rest/api/3/issue",
     "method": "POST",
     "auth_user": "alerts@mycompany.com",
     "auth_password": "<jira-api-token>",
     "body": "{\"fields\": {\"project\": {\"key\": \"OPS\"}, \"summary\": \"{message}\", \"issuetype\": {\"name\": \"Bug\"}}}"
   }
 }'

Test the alert with a custom message:
```
cw-alertctl -i jira_bug_node_work_queue test --message "Work queue test" --send
```
If everything is configured properly, the alert sends the test message to Jira and creates a bug with the test message text.

Example: Severity-based Alert to Multiple Targets#

The following complex example configures alerts for Slack and email. The alert sends to Slack when the severity is WARNING or higher and to email when the severity is DEBUG or INFO. Multiple mutators are used to transform the original MQTT event message and set the severity level.

Create a mutator with some of the MQTT topics related to ARS work queue as the sources. The mutator maps the upstream ARS event fields to new fields and sets the severity field to WARNING.
```
cw-mutatorctl create --content '{
  "name": "slack_transform_work_queue_message",
  "description": "Transforms the Move to Work Queue MQTT message for Slack",
  "sources": [
      "mqtt:ars/v1/+/events/provisioning_error",
      "mqtt:ars/v1/+/events/drain_failure",
      "mqtt:ars/v1/+/events/require_manual_fix"
  ],
  "format": "{\"node\": \"{node}\", \"detail\": \"{message}\", \"time\": \"{when}\"}",
  "severity": "WARNING"
}'
```
The Provisioning Error, Drain Failure, and Require Manual Fix state transitions are automated and occur when ARS detects an issue with a compute node and sends the node to Work Queue. Using a higher severity for these topics draws attention to nodes that require admin intervention in Work Queue as soon as the state change occurs.
Create a mutator with the Require Human MQTT topic as the source that maps the upstream ARS event fields to new fields and sets the severity field to INFO.
```
cw-mutatorctl create --content '{
  "name": "email_transform_work_queue_message",
  "description": "Transforms the Move to Work Queue MQTT message for email",
  "sources": ["mqtt:ars/v1/+/events/require_human"],
  "format": "{\"node\": \"{node}\", \"detail\": \"{message}\", \"time\": \"{when}\"}",
  "severity": "INFO"
}'
```
The Require Human state transition is used when a node is manually moved to the Work Queue state when an admin issues a Manual Drain request. As a result, the move to Work Queue is expected and therefore the severity level is not as high as other Work Queue state transitions.

Create a mutator with the Slack and email mutators as the sources. The new mutator formats the text from either source into a human-readable message. It does not change the severity of the original messages.

cw-mutatorctl create --content '{
  "name": "human_readable_work_queue_message",
  "description": "Makes the Move to Work Queue message human-readable",
  "sources": [
    "mutator:slack_transform_work_queue_message",
    "mutator:email_transform_work_queue_message"
  ],
  "format": "{\"message\": \"{node}: {detail} at {time}\"}"
}'

Determine the required fields for Slack:

[admin@head1 ~]$ cw-alertctl schema slack
{
  "required": [
    "webhook_url",
    "message"
  ]
}

Determine the required fields for email:

[admin@head1 ~]$ cw-alertctl schema email
{
  "required": [
    "message",
    "to"
  ]
}

Create an alert for Slack that references the human-readable mutator and sends when the severity is WARNING or higher.
```
cw-alertctl create --content '{
   "name": "slack_node_to_work_queue",
   "description": "Send alert to Slack for nodes in work queue with severity WARNING or higher",
   "sources": ["mutator:human_readable_work_queue_message"],
   "output": "slack",
   "output_config": {
     "webhook_url": "https://hooks.slack.com/services/ABCD",
   "channel": "#warning"
   },
   "min_severity": "WARNING"
 }'
```
Only messages that originate from the slack_transform_work_queue_message mutator and related MQTT topics are processed because they have a severity set to WARNING. Messages from email_transform_work_queue_message are dropped because they have a severity set to INFO and are filtered out.

Create an alert for email that references the human-readable mutator and sends when the severity is set to DEBUG or INFO, but not any higher severities.

cw-alertctl create --content '{
   "name": "email_node_to_work_queue",
   "description": "Send alert to email for nodes in work queue with low severity",
   "sources": ["mutator:human_readable_work_queue_message"],
   "output": "email",
   "output_config": {
     "to": "name@example.com",
     "from": "notifiers@head1",
     "host": "example-com.mail.protection.outlook.com",
     "port": 25,
     "tls": true,
     "subject": "Alert: ClusterWareAI Node Entered Work Queue"
   },
   "severities": ["DEBUG", "INFO"]
 }'

Only messages that originate from the email_transform_work_queue_message mutator and related MQTT topic are processed because they have a severity set to INFO. Messages from slack_transform_work_queue_message are dropped because they have a severity set to WARNING, which does not match INFO or DEBUG.

If everything is configured properly, events with a severity of WARNING or higher are sent to the Slack channel associated with the webhook. Events with a severity of DEBUG or INFO are sent to a specified email. Events without any severity set are not sent to either output target.