Alerts#

An alert defines which ClusterWareAI ™ events are monitored, how those events are processed, the notification severity assigned to the resulting alert, where notifications are delivered, and whether the alert is currently enabled or muted. You can create alert definitions that monitor specific event sources and deliver notifications when those events occur using the cw-alertctl tool.

Alert Severity#

Alerts support severity-based routing and filtering to help distinguish informational events from conditions that require immediate attention. Use severity levels to control which events are delivered to specific audiences or external systems. Supported severity levels include DEBUG, INFO, WARNING, ERROR, and CRITICAL (in order from least to most severe). For example, an INFO alert might be delivered to a general email address while an ERROR alert might be directed to a closely monitored Slack channel.

Severity is optional and can come from multiple sources. MQTT topics, description fields, and messages can have a severity setting. Severity can be adjusted or set by a mutator. You can configure alerts to send if a severity level matches the configuration and to not send when the severity differs. Alert severity fields include:

  • severity: Sets the severity added to the alert log when the incoming message from MQTT or a mutator does not have a severity set. The value is not set on the alert message itself, only in the alert log. If the incoming message has a severity value, the incoming message severity value is used in the log instead.

  • severities: List of severity values that cause the alert to send. For example, setting "severities": ["WARNING", "ERROR"] causes the alert to send for messages with a severity of WARNING or ERROR, but not for INFO, DEBUG, or CRITICAL. This field cannot be set with min_severity.

  • min_severity: Severity value threshold at which an alert sends. The alert sends at this severity level or higher. For example, setting "min_severity": "ERROR" causes the alert to send for messages with a severity of ERROR or CRITICAL, but not any other value. This field cannot be set with severities.

Setting severities or min_severity is useful when you have different output targets based on message severity. For example, send INFO and DEBUG alerts to email, WARNING messages to Microsoft Teams, and ERROR or higher to PagerDuty. See Example: Severity-based Alert to Multiple Targets for a detailed example.

Supported Alert Targets#

The ClusterWareAI software supports sending alerts to the following third-party software targets:

  • Email

  • PagerDuty

  • Slack

  • Microsoft Teams

  • Telegram

You can also send alerts via a generic webhook endpoint. This supports additional targets, such as ServiceNow and Jira.

Other third-party targets are supported by the underlying alerting library, but may not be fully supported by the ClusterWareAI software. Use cw-alertctl schema for a full list of possible targets. Contact Penguin Solutions if you encounter issues when using one of the alternative targets not listed above.

Default Alerts#

The ClusterWareAI software creates a default alert ars_work_queue_log during the install process. The default alert uses the default mutator ars_work_queue as the source and outputs the alert to the alerting log. To update the alert target to a monitored location, such as email, clone the default alert and update the output and output_config to match the new output location.

Restore the default alert using the cw-alertctl --ensure-defaults command if needed.

Muting Alerts#

Alerts can be muted for a configurable amount of time to allow you to address an ongoing issue. For example, if you have a network outage, you may receive repeat alerts until the issue is resolved. Muting also can be useful if your cluster has a planned downtime that could cause multiple alerts to send.

For example, to mute an alert for three hours:

cw-alertctl -i outage_alert_slack update mute_for=3h

To remove an existing mute from an alert:

cw-alertctl -i outage_alert_slack update mute=false

Alert History#

Alerting maintains a record of generated alerts and processing activity for visibility into previous alerts. Use the cw-reports alerts tool to view logs of previous alerts, including whether sending the alert was successful. Refine the list of alerts using the --nodes, --since, and --severity options to find alert logs that match your criteria. See cw-reports for details.

Note

In a cluster with multiple head nodes, the database stores alert activity specific to that head node. It does not share data with other head nodes.