Running ClusterWareAI in a Container on Compute Nodes#

In a typical cluster configuration, ClusterWareAI ™ health checks, Telegraf metrics, and status indicators run directly on the compute node. In some cases, enterprise customers may want to run health checks and other tools within a container on a compute node for security purposes. The ClusterWareAI software provides a container image you can run via Podman or Kubernetes container runtimes. The ClusterWareAI compute node container uses Red Hat Universal Base Image (UBI) 9 minimal.

Health checks are deployed as a single binary called scout on ClusterWareAI compute nodes using containers. The scout binary determines which health checks apply to a node and runs those checks on a scheduled basis. In a typical cluster, the health checks are applied to a compute node using the _ars_checks reserved attribute. In a cluster with containerized compute nodes, scout creates a configuration file that is used to specify relevant health checks.

Configuration requirements differ for Kubernetes and Podman.

Configure Kubernetes Compute Node Container#

Prerequisites:

You should have ClusterWareAI installed and Kubernetes configured for cluster with your compute nodes set up as worker nodes. See Configure Kubernetes for details.
You need your authentication code (cluster serial number) and password (if applicable) from Penguin Solutions to access repo.ice.penguinsolutions.com.

Configure Environment and Kubernetes#

Install the health check DaemonSet with Helm. These steps pin a version-specific chart, which deploys image repo.ice.penguinsolutions.com/clusterware-node-ubi:<version>.

Replace placeholder scheduler URLs and credentials for the target environment.
Set names. Namespace, release, and override file path are operator-chosen. SECRET and DS are derived from the release name (DS is the chart fullname). VER is the latest published chart (13.3.0-g0001 in the example). The published chart already stamps the image tag.

For example:
```
export NS=health-scheduler          # operator-chosen namespace
export RELEASE=health-scheduler     # operator-chosen release name
export SECRET="${RELEASE}-registry"
export DS="${RELEASE}-clusterware-node-ubi"
export VALUES=/tmp/clusterware-node-ubi-override.yaml
export VER=13.3.0-g0001
```
Subsequent commands require these variables. Export them in each new shell session.
Create a Kubernetes namespace and label it with baseline:
```
kubectl create namespace "$NS"
kubectl label namespace "$NS" \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/audit=baseline \
  pod-security.kubernetes.io/warn=baseline
```
The chart can pass Baseline when hostNetwork stays unset and host /sys / /dev stay unmounted. Do not add capabilities (including SYSLOG). Do not set privileged: true. Restricted is not supported (capabilities.drop: [ALL] breaks the agent). A later step sets uidMode=auto so the process is non-root.
Set environment variables by exporting the repo username and password obtained from Penguin Solutions:
```
export REPO_USER='<AUTHENTICATION_CODE>'
export REPO_PASS='<PASSWORD>'
```
If you do not have a password, set REPO_PASS='registry'.

Authenticate Helm to repo.ice.penguinsolutions.com with your access details via stdin:

printf '%s' "$REPO_PASS" | helm registry login repo.ice.penguinsolutions.com \
  --username "$REPO_USER" --password-stdin

Create the pull secret in the release namespace so pods can pull images from the repo:

kubectl --namespace "$NS" create secret docker-registry "$SECRET" \
  --docker-server=repo.ice.penguinsolutions.com \
  --docker-username="$REPO_USER" \
  --docker-password="$REPO_PASS"

Install the Helm Chart#

There are two options when installing the helm chart: using an override file or providing options via command line arguments.

The following table defines available options and values.

Field	Description	Acceptable Values
`existingSecret`	The name of the pre-created Docker pull secret Kubernetes object. This option is useful when re-deploying and upgrading the release.	Set from `$SECRET`
`scout log level`	Output generated by Scout binary execution. Defaults to info. Debug mode is more verbose.	info, debug
`status`	Determines if status updates are sent to the head node. If set to false, the compute node will have a Down status on the head node and you will not be able to update node attributes. Defaults to True if not specified. See Command-Line Monitoring of Nodes for details.	true, false
`telegraf`	Determines if Telegraf runs on the compute node and sends metrics to the head node and Grafana. Defaults to True if not specified. See Grafana Telemetry Dashboard for details.	true, false
`base_url`	The head node that the compute node sends data to. For example, if health checks are enabled, this is the head node that receives pass/fail messages. Defaults to the parent head node if not specified.	Head node IP address
`checks`	The health checks and health check labels to be applied to the compute node. Health checks and health check labels must exist in the ClusterWareAI software. Health checks are specified by name and health check labels use a percent sign (%) prefix, just as they are when using the `_ars_checks` reserved attribute. Include the argument multiple times if you want to include a list of checks and check labels. Defaults to the `_ars_checks` reserved attribute value if not specified. `nodeconfig.sched.checks` must be a list, not a string. See Add Health Checks to Nodes for details.	Health check name, health check label name with % prefix
`checks_bundle_url`	Used with the checks argument. Location of the Scout binary file. Defaults to the parent head node if not specified.	ClusterWareAI API, file:// URL, absolute file path
`checks_definition_url`	Used with the checks argument. Location of the JSON file that defines the health checks. Defaults to the parent head node if not specified.	ClusterWareAI API, file:// URL, absolute file path

Option 1: Use an Override File#

Write the override file from a heredoc. Replace scheduler URLs in the heredoc for the target environment. Head URL and checks are supplied in sched.yaml (nodeconfig.sched.*).

cat > "$VALUES" <<EOF
registry:
  existingSecret: ${SECRET}

nodeconfig:
  scout:
    logLevel: info
  sched:
    status: false
    telegraf: false
    base_url: "https://<CW HEAD NODE IPV4 ADDR>/api/v1"
    checks_definition_url: "file:///opt/cw/assets/checks.json"
    checks_bundle_url: "file:///usr/local/bin/scout"
    checks:
      - "%default"

security:
  uidMode: auto

podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
EOF

Note

Values given for security, podSecurityContext, and securityContext are required should not be modified.

Install the pinned chart version:

helm install "$RELEASE" \
  oci://repo.ice.penguinsolutions.com/charts/clusterware-node-ubi \
  --version "$VER" --namespace "$NS" --timeout 15m --wait \
  -f "$VALUES"

In the example, the following is configured in the namespace:

Compute node status reporting is disabled
Telegraf metrics are disabled
Set up head.example\:8080 for health check reporting
Health check definitions are in a local file checks.json
The Scout binary is available from a local directory /usr/local/bin/scout
All health checks with the default label will run

Option 2: Use Command Line Arguments#

Alternatively, instead of using an override file, pass the values as --set flags:

helm install "$RELEASE" \
oci://repo.ice.penguinsolutions.com/charts/clusterware-node-ubi \
  --version "$VER" --namespace "$NS" --timeout 15m --wait \
  --set registry.existingSecret="$SECRET" \
  --set nodeconfig.scout.logLevel=info \
  --set nodeconfig.sched.status=false \
  --set nodeconfig.sched.telegraf=false \
  --set-string nodeconfig.sched.base_url="https://<CW HEAD NODE IPV4 ADDR>/api/v1" \
  --set-string nodeconfig.sched.checks_definition_url="file:///opt/cw/assets/checks.json" \
  --set-string nodeconfig.sched.checks_bundle_url="file:///usr/local/bin/scout" \
  --set-string 'nodeconfig.sched.checks[0]=%default' \
  --set security.uidMode=auto \
  --set podSecurityContext.runAsNonRoot=true \
  --set podSecurityContext.seccompProfile.type=RuntimeDefault \
  --set securityContext.allowPrivilegeEscalation=false

Note

Values given for security, podSecurityContext, and securityContext are required should not be modified.

Validate Configuration#

After Scout is set up, a compute node configuration file is saved to /opt/cw/nodeconfig/sched.yaml. The settings in the configuration file supersede any settings from the head node. For example, if you provide a container configuration that includes health checks and also set the _ars_checks reserved attribute on the same compute node within the ClusterWareAI software, the health checks listed in the reserved attribute are ignored and only the health checks applied via the container configuration are run on the compute node. This means the ClusterWareAI head node will not have visibility into which health checks are running on each compute node, how often they are running, and so on. Health check success and failure messages are still sent to the head node. See Health Check Data Flow for details.

To verify the configuration:

List the DaemonSet, health check pods, and secrets, wait until the health check DaemonSet is rolled out, and confirm the image matches the version number. Confirm the namespace is baseline, the pod spec has no hostNetwork or hostPath, and the process is non-root.:

kubectl --namespace "$NS" get daemonset,pods,secret
kubectl --namespace "$NS" rollout status "daemonset/$DS"
kubectl --namespace "$NS" get "daemonset/$DS" \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl get namespace "$NS" --show-labels
kubectl --namespace "$NS" get "daemonset/$DS" \
  -o jsonpath='hostNetwork={.spec.template.spec.hostNetwork}{"\n"}'
kubectl --namespace "$NS" get "daemonset/$DS" \
  -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" hostPath="}{.hostPath.path}{"\n"}{end}'
helm get values "$RELEASE" --namespace "$NS"
kubectl --namespace "$NS" get pods -l app.kubernetes.io/instance="$RELEASE" \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.spec.securityContext.runAsUser}{"\n"}{end}'
kubectl --namespace "$NS" exec "daemonset/$DS" -- id -u
kubectl --namespace "$NS" get "daemonset/$DS" \
   -o jsonpath='{.spec.template.spec.securityContext}{"\n"}'
kubectl --namespace "$NS" get "daemonset/$DS" \
  -o jsonpath='{.spec.template.spec.containers[0].securityContext}{"\n"}'

helm get values must show security.uidMode: auto and podSecurityContext.runAsNonRoot: true. id -u must match runAsUser in 10000–65533 (not 0). Container securityContext must show allowPrivilegeEscalation: false.

Exec into a health check pod and verify that the scout binary and /opt/cw/nodeconfig are present:

kubectl --namespace "$NS" exec "daemonset/$DS" -- \
  scout --version
kubectl --namespace "$NS" exec "daemonset/$DS" -- \
  ls -la /opt/cw/nodeconfig

Run the chart test and display logs. The helper checks scout.yaml / sched.yaml contents and scout --version. root on the projected volume listing is the volume owner, not the container UID.
```
helm test "$RELEASE" --namespace "$NS" --logs
```
Run a health scan (scout scan) from the deployed DaemonSet or from one of its pods:
```
kubectl --namespace "$NS" exec "daemonset/$DS" -- \
  scout scan -exit-zero
```

Update Kubernetes Compute Node Container Configuration#

If you want to modify your compute node container configuration after initial setup, use the helm upgrade command. Compute nodes do not need to be restarted as the associated DaemonSet roll out updated pods on each worker node sequentially.

Auto Remediation Service (ARS) and Containerized Health Checks#

ARS can still be used to automatically remediate health check failures on nodes that use containers. To enable ARS on compute nodes that use containers:

Assign health checks with compute nodes as described in Install the Helm Chart.
Ensure the health checks assigned to the compute nodes have associated remedies. See Remedies and Actions for details.
Enable ARS on the compute nodes using the instructions in Configure ClusterWareAI Health Monitoring System and Auto Remediation Service. You do not need to enable CHMS on the compute nodes and can proceed directly to the Enable ARS on Compute Nodes section.