Known Issues And Workarounds#

The following are known issues of significance with the latest ClusterWareAI ™ version and suggested workarounds.

  • If you are updating from version 13.2 to version 13.3 or later and you are using health checks on your compute nodes, you must update your compute node images and packages at the same time as your head nodes. Delaying the update to the compute nodes can cause the compute nodes to run all health checks instead of only those that are assigned, which leads to false failures.

  • The clone action is not currently supported for remedies. Create a new remedy or update an existing remedy instead.

  • The tool cw-kube does not currently support installations of Kubernetes version 1.34 and later on RHEL 8 systems. If you are using RHEL 8 and Kubernetes, run the following command when configuring Kubernetes after installing 13.3.0 or updating to version 13.3.0 from an earlier version:

    cw-kube -i <control plane node ID> --init --version 1.31.2
cw-kube -i n[<node IDs>] --join --cluster <control plane node ID> --version 1.31.2

  • If you have installed the optional Software Defined Networking (SDN) package and are using the AI Factory Operations Agent to query switch data, agent visibility into topology data is limited.

  • The AI Factory Operations Agent cannot provide live ARS or CHMS information for a specific node. It can provide historical ARS data, such as the ARS history or state transition history of a node.

  • Slurm version 23.02.7 cannot be directly updated to Slurm version 25.05.4 while preserving the existing Slurm accounting database. If you need to maintain your Slurm database between these versions, contact Penguin Solutions for assistance.

  • If you are using ClusterWareAI with podman, switching between users can lead to podman errors. Using SSH to connect as the user typically solves the problem.

  • Rocky 10 images are not supported with SELinux enforcing. Use permissive mode instead.

  • Rocky 10 images cannot be deployed with Ignition.

  • Using Ignition to boot compute node images with Oracle 8 or Oracle 9 is not supported.

  • Using Ubuntu 20.04 for images is not currently supported.

  • The ContainerRuntime default health check assumes your Kubernetes cluster is configured to read the /var/lib/kubelet/kubeadm-flags.env file. If this file does not exist for your cluster, or if the file cannot be read, then the ContainerRuntime health check throws an error when it runs on worker nodes. To resolve the issue, remove the health check from worker nodes or enable read on the file, if available.

  • If administrators are using cw-modimg to concurrently modify two different images, then one administrator will see a message similar to:

    WARNING: Local cache contains inconsistencies.
Use --clean-local to delete temporary files, untracked files,
and remove missing files from the local manifest.

    then use cw-modimg --clean-local.

    However, only execute --clean-local after all cw-modimg image manipulations have completed.

  • Ensure that /etc/sudoers does not contain the line Defaults requiretty; otherwise, DHCP misbehaves.

  • The NetworkManger-config-server package includes a NetworkManager.conf config file with an enabled "no-auto-default" setting. That is incompatible with ClusterWareAI compute node images and will cause nodes to lose network connectivity after their boot-time DHCP lease expires. Either disable that setting or remove the NetworkManger-config-server package from compute node images.

  • The cw-clusterctl repos create command has a urls= argument that specifies where the new repo's contents can be found. The most common use is urls=http://<URL>. The alternative urls=file://<pathname> is not currently supported. Instead, you must first manually create an http-accessible repo from that pathname. See Creating Local Repositories without Internet.

  • When moving a head node from one etcd-based cluster to another using the managedb join command, reboot the joining head once the join is complete.

  • If a new head node is failing to join an existing etcd-based cluster, check /var/log/clusterware/etcd.log and look for repeated lines of the form:

    <DATE> <SERVER> etcd: added member <HEX> [<URL>:52380] to cluster <HEX>

    If the log file contains multiple of these lines per join attempt, then try running managedb recover on an existing head node and joining all head nodes back into the cluster one at a time. Re-joining heads that were previously in the cluster may require a --purge argument, such as managedb join --purge <IP>

  • The Grafana service can be missing from the list of services on one or more head nodes in a multi-head node cluster. To resolve the issue:

    1. Make sure the Grafana service is running:

      systemctl status grafana-server

    2. If the service is running, restart ClusterWareAI on the head node(s) that are missing the service.

    3. If the service is not running, check the log files and troubleshoot the grafana-server service.

  • If you are using STIG and the Embedded Registry fails to start, stop the fapolicyd service, start the cw-embedded-registry service, then restart fapolicyd.

  • cw-install performs its early check to determine if a newer clusterware-installer RPM is available by parsing the appropriate clusterware repo file (typically /etc/yum.repos.d/clusterware.repo) to find the first base_url= line. If there are multiple such lines, (such as specifying multiple ClusterWareAI repos), then the cluster administrator should order the repos so that the repo containing the newest RPMs is the first repo in the file.

  • The following error may appear during ClusterWareAI installation: Error occurred in X509_REQ_set_version: [error:05880106:x509 certificate routines::passed invalid argument]. Make the following changes before attempting installation again:

    1. Update the openssl and sscg packages to their latest versions.

    2. Remove the following files if they are present and empty:

      • /etc/pki/tls/certs/localhost.crt

      • /etc/pki/tls/private/localhost.key

    3. Run cw-install --clear-all

    4. Re-attempt the ClusterWareAI install.

  • The ClusterWareAI install may not push images to the container registry, which can cause issues using the clusterware-node image. To fix the issue, re-run /opt/scyld/clusterware/bin/push-images.

  • After updating from version 13.0.0 to version 13.2.0 or later, the grafana-server service may not start.

    1. Run systemctl status grafana-server to see if the service started.

    2. If the service did not start, you can either run the commands below or restart the head node. Running the commands is less impactful on your cluster.

      Run the following commands:

      kill -9 $(pidof grafana)
rm -f /run/grafana/grafana-server.pid /var/run/grafana/grafana-server.pid
systemctl restart grafana-server

    3. Run systemctl status grafana-server again to verify the service started.

  • After updating from version 13.0 or earlier, booting using iscsi does not work by default. Install the iscsi-initiator-utils package inside the image and recreate the initramfs after updating the image.

  • After updating from version 12 to version 13, stateless nodes may fail to boot with errors in the console of the booting node that mention attributes. To solve the problem, update the initramfs of the affected nodes by running cw-mkramfs --update <boot_config>

  • When using Ignition to make partitions, the partition specified with _disk_root will be formatted with the ext4 file system even if the ignition file specifies another file system such as XFS.

  • When booting images based on Linux distributions such as Ubuntu 24.04 (Noble Numbat), the compute node may fail to boot with a "Image too large to unpack into /sysroot" message displayed on the system terminal. This is caused by a newer version of the umount command. Upgrade to version 13.0 or later or contact Penguin Solutions for a patch that can be applied to earlier versions.