Add CHMS and ARS to Tenancy

Note

Tenants, tenancies, and other features are available if you have paid for multi-tenant support. Contact Penguin Solutions to learn more.

After the tenancy is created, configure the ClusterWareAI Health Monitoring System (CHMS) and Auto Remediation Service (ARS) in the new tenancy.

1. Determine which health checks or health check labels are applicable to the tenancy configuration and to the tenant's needs. See View Health Checks for options.

2. From the tenancy head node, create a new attribute group:

   cw-attribctl create name=<group name>
   

3. Set the _ars_state, _ars_scheduler, and _ars_checks reserved attributes on the new attribute group:

   cw-attribctl -i <group name> set _ars_state=new_node _ars_scheduler=slurm \
     _ars_checks=<health checks>
   

   Where <health checks> is replaced by the list of health checks and health check labels you selected for the tenancy.

4. Join all tenancy compute nodes to the new attribute group:

   cw-nodectl -i <node list> join <group name>
   

   The compute nodes should power on and begin running health checks. The compute nodes are ready for use when they are in the Available state.

Compute nodes in a tenancy that fail health checks are remediated by the tenancy head node. The tenancy administrator can monitor health check and ARS data in the tenancy ClusterWareAI instance and handle any manual fixes for nodes in the Work Queue. See Auto Remediation Service (ARS) for details.

By default, health check failure and ARS alerts are not sent to the super-cluster. Tenancy compute nodes that fail health checks consistently should be moved to the Quarantine tenancy for further testing by the superadministrator. See Add Nodes to Quarantine Tenancy for details.