State Maps

The ClusterWareAI ™ node selection language identifies nodes that match possibly complex criteria by matching attribute values, detected status, or hardware details using basic comparators and logical operators. However, polling the ClusterWareAI service for attribute status frequently or across many nodes is inefficient and can impact the head node performance in some cases. To alleviate this, the ClusterWareAI platform provides state maps to identify a node or groups of nodes by a defined selector.

State maps provide a general purpose mechanism to select groups of nodes based on their status and configuration. You can watch for changes in the groups to trigger actions. Use a state map to track nodes through scenarios including:

• Error detection, handling, and requalification

• A rolling firmware update process

• Idle-time performance testing

Temporary and Persistent State Maps

The cw-nodectl waitfor mechanism creates a state map based on a state expression provided in the command. Temporary state maps are not named. They can be extended when in use, but otherwise expire and are deleted after 10 seconds of inactivity. In a multi-head cluster, temporary state maps are only available on the head node running the command, so all related commands must be passed through the same head node.

One common use for a temporary state map is to wait for a node to boot before proceeding with additional steps in an overall command. For example:

cw-nodectl -in10 reboot then waitfor up then exec uptime

The up in the example command is shorthand for a longer selector, specifically status[state] == “up” and can be replaced with more complicated selectors. For example, if the administrator is not rebooting the node but executing a command that will modify a node attribute when it completes.

Command chaining with “then” allows for simple automation, but more complex automation deals with multiple nodes at different stages. For that case, you can provide a set of selectors as a persistent state map using cw-clusterctl statemaps. Persistent state maps are named and can be activated on one or more head nodes in a multi-head cluster. They can be used once or many times based on need. Once activated, commands related to the state map, such as identifying nodes in each state, can only be run on the head node that activated the state map.

Default State Maps

There are two persistent state maps included with the ClusterWareAI software at installation. The default state maps are frozen and should not be modified or renamed.

Name	Description	Selector
NodeStatus	Groups nodes by current up/down/ booting status. Used by the ClusterWareAI GUI.	booting, down, up
ARS	Groups nodes by Auto Remediation Service (ARS) status. While this state map is available for all clusters, it only includes compute nodes with ARS enabled. Used by the Remediation State Machine.	new node, provisioning, available, drained, draining, auto remediation, work queue

State Map Activation

State maps typically expire after 10 seconds of inactivity to save resources. If the state map is temporary, the state map is removed after expiring. If the state map is persistent, the state map is deactivated after expiring. The cw-nodectl waitfor command automatically extends the expiration until all data is returned. You can also extend state map expiration using a PUT /nodes/waitfor/<STATE MAP NAME> call with a duration parameter. You cannot decrease the state map activation duration.

Use the cw-clusterctl statemaps tool to activate a state map. For example:

cw-clusterctl statemaps -i my_states activate

You can also use the tool to view the current state map activation status, activation duration, and head node where active (if any). For example:

$ cw-clusterctl statemaps -i status ls -l
Statemaps
status
  active: True
  active_on_head: head1
  expires_in: 80
  last_modified: 2026-04-23 11:59:14 UTC (0:03:37 ago)
  last_modified_by: admin1
  last_modified_on: head1
  name: status
  states
    booting: status[state] == "booting"
    down: status[state] == "down"
    up: status[state] == "up"

Create State Map

1. Use the cw-clusterctl statemaps tool or the State Maps Page in the ClusterWareAI GUI to create a persistent state map. For example:

   cw-clusterctl statemaps create name=my_states \
     states='{"redstate":"index > 10","bluestate":"index > 100"}'
   

   This creates a state map with two states.

2. Review the state map details in the ClusterWareAI GUI or by running:

   $ cw-clusterctl statemaps ls -L
   Statemaps
    my_states
      active: False
      last_modified: 2026-03-02 20:43:46 UTC (0:00:24 ago)
      last_modified_by: admin1
      last_modified_on: head23.cluster.local
      name: my_states
      states
        bluestate: index > 100
        redstate: index > 10
   

3. Activate the state map:

   cw-clusterctl statemaps -i my_states activate
   

4. List the nodes that match the selectors:

   $ cw-clusterctl statemaps -i my_states nodelist
   Nodes
     n[0-5]: bluestate
     n[6-15]: redstate
   

You can create complex state maps using logical operators. For example, you can update the previous state map to identify nodes with a certain index number range and boot style:

cw-clusterctl statemaps -i my_states update states=\
  '{"redstate":"index > 10" AND "attributes[_boot_style]==\"rwram\"",\
  "bluestate":"index > 100" AND "attributes[_boot_style]==\"rwram\""}'

State maps can be frozen to avoid accidental changes. For example:

cw-clusterctl statemaps -i my_states update frozen=True

Load Example State Map

An example state map is provided as part of the clusterware-tools package:

$ cat /opt/scyld/clusterware-tools/examples/node-states.ini
[status]
up = status[state] == "up"
down = status[state] == "down"
booting = status[state] == "booting"

This INI format defines a state map named “status” containing 3 states named “up”, “down”, and “booting”. The selector that defines each state is provided to the right of the name, after the equal sign.

This file can also be written in JSON format:

{
  "name": "status",
  "states": {
      "up": "status[state] == \"up\"",
      "down": "status[state] == \"down\"",
      "booting": "status[state] == \"booting\""
  }
}

1. Load the state map through the cw-nodectl command:

   cw-nodectl waitfor --load @node-status.json
   

2. Use the waitfor command to activate the loaded state map by name and list nodes that match the selectors:

   $ cw-nodectl waitfor --name status
   Nodes
     n[5-8,10]: up
   

3. Additional arguments allow for streaming state transitions or simplifying the output for easier parsing:

   $ cw-nodectl waitfor --stream --name status
   n[1-4] in down
   n[5-10] in up
   
   n[5] left up
   n[5] entered down
   n[5] left down
   n[5] entered booting
   n[5] left booting
   n[5] entered up
   

   In the example, a single node (n5) in a 10 node cluster was rebooted and state transitions were emitted as the node progressed from “up” to “down” to “booting” and back to “up”.

View Nodes in State Map States

There are multiple ways to see which nodes match each state map state in a persistent (named) state map.

• Via the cw-nodectl waitfor command. For example:

  cw-nodectl waitfor --name <STATE MAP NAME>
  

• Using the cw-clusterctl statemaps command to activate the state map and get a list of applicable nodes. For example:

  cw-clusterctl statemaps -i <STATE MAP NAME> activate
  cw-clusterctl statemaps -i <STATE MAP NAME> nodelist
  

• Using the waitfor or statemap API to activate the state map and get a list of applicable nodes. For example:

  PUT /nodes/waitfor/<STATE MAP NAME>
  GET /nodes/waitfor/<STATE MAP NAME>/nodes
  
  PUT /statemap/<STATE MAP NAME>/activate
  GET /statemap/<STATE MAP NAME>/nodes
  

• Using the API to activate the state map and MQTT to get a list of applicable nodes.

• For the default ARS state map, using the ARS State Machine Page in the ClusterWareAI GUI.

On a multi-head cluster, node data is only available from the head node that activated the state map. You can activate the same state map on multiple head nodes if desired.