Mutators#

A mutator transforms event data before it is delivered as an alert. Mutators can convert raw MQTT event data into messages that are easier for recipients to understand and act upon. Mutators can also be chained together, allowing multiple transformations to be applied before an alert is delivered. Chaining mutators enables you to tailor alert content for different audiences and delivery targets.

Use the cw-mutatorctl tool to create and manage mutators.

Default Mutators#

The ClusterWareAI ™ software creates a default mutator ars_work_queue during the install process. The default mutator uses MQTT topics that occur when compute nodes move into the ARS Work Queue state as sources. The default mutator is referenced by the default alert ars_work_queue_log.

Restore the default mutator using the cw-mutatorctl --ensure-defaults command if needed.

Mutator Severity#

Mutators can optionally set or adjust a severity value from an MQTT message or another mutator. MQTT topics, description fields, and messages can have a severity setting. You can configure mutators to pass along messages if a severity level matches the configuration and not when the severity differs. Supported severity levels include DEBUG, INFO, WARNING, ERROR, and CRITICAL (in order from least to most severe). Mutator severity fields include:

  • severity: Assigns the severity to all messages that pass the filter.

  • severity_from_field: Extracts the severity value from a dotted path in the message if the field contains a known code (INFO, DEBUG, and so on). For example, label or description.severity.

  • severity_from_topic: Extracts the severity from the MQTT topic or a passed-down topic value if the topic has a known code (INFO, DEBUG, and so on). For example, the MQTT topic ars/v1/cluster/events/DEBUG/node results in a severity of DEBUG.

  • severities: List of severity values that cause the mutator to process. For example, setting "severities": ["WARNING", "ERROR"] causes the mutator to process messages with a severity of WARNING or ERROR, but not messages with INFO, DEBUG, or CRITICAL. This field cannot be set with min_severity.

  • min_severity: Severity threshold at which the mutator processes messages. The mutator processes messages at this severity level or higher. For example, setting "min_severity": "ERROR" causes the mutator to process messages with a severity of ERROR or CRITICAL, but not any other value. This field cannot be set with severities.

Important

If you plan to use severity_from_topic or severity_from_field, do not use any severity values in your primitive names (compute nodes, DPUs, attribute groups, and so on) to avoid incorrect severity assignments. For example, DEBUG_NODE matches DEBUG.

Mutator Severity Resolution#

Mutator severity is resolved in a specific order. When a message from MQTT or another mutator arrives, severity resolves in the following order:

  1. Existing severity on the message from MQTT or another mutator.

  2. severity_from_field if the path results in a valid code. If multiple values are in the same field, the most severe value is used.

  3. severity_from_topic if the message is from an MQTT topic or a passed-down topic value and contains a valid severity value. If multiple values are in the same topic, the most severe value is used.

  4. Filter the message based on severities or min_severity values. If the incoming message does not have a severity set and either severities or min_severity is specified, the message is dropped.

  5. Assign the value specified by the mutator's severity field.

Example: Mutator Severity#

The following mutators process a message originally from the ars/v1/+/events/provisioning_error MQTT event. The latter two mutators configure and filter severity values.

Note

Some required fields are omitted for simplicity.

{
  "name": "ars_prov_error",
  "description": "Node failed provisioning and is sent to the work queue",
  "sources": ["mqtt:ars/v1/+/events/provisioning_error"],
  "format": (
      '{"when": "{when}", "node": "{node}", "event": "{event}", '
      '"data": "{node}: {event}: {message}", "topic": "{topic}"}'
  )
}

{
  "name": "ars_prov_error_from_field",
  "description": "Sets severity based on data field",
  "sources": ["mutator:ars_prov_error"],
  "severity_from_field": "data",
  .
  .
  .
}

{
  "name": "ars_prov_error_min_severity",
  "description": "Filters messages based on severity WARNING or higher",
  "sources": ["mutator:ars_prov_error"],
  "min_severity": "WARNING",
  .
  .
  .
}

In this example:

  • The ars_prov_error mutator outputs a message without a severity value set. The message is received by both the ars_prov_error_from_field and ars_prov_error_min_severity mutators.

  • The ars_prov_error_from_field mutator outputs a message with a severity value set to ERROR because the data field it scans contains the name provisioning_error (for example, n1: provisioning_error: disk not found). Matching severity values do not need the same capitalization.

  • The ars_prov_error_min_severity mutator drops the message because the message it receives from the ars_prov_error mutator does not have a severity value and is filtered out.

Mutator Graph#

Use the cw-mutatorctl graph action to visualize relationships between mutators, MQTT topics, and other mutators. You can also use the graph action to avoid accidental loops if you have multiple mutators.

For example:

[admin@head ~]$ cw-mutatorctl graph
mqtt:/clusterware/v1/api/bootconfigs
`-- mutator:bootconfig_event_message
mqtt:/clusterware/v1/api/images
|-- mutator:image_audit_fields
|   `-- mutator:image_notify
`-- mutator:image_event_message
    `-- mutator:image_notify (see above)

The example output shows that a single mutator is associated with the /clusterware/v1/api/bootconfigs MQTT topic and two sets of mutators are associated with the /clusterware/v1/api/images topic and have inheritance relationships with other mutators.