DPUs#

The ClusterWareAI ™ software supports data processing units (DPUs) associated with existing compute nodes. DPUs are managed using the cw-dpuctl tool or DPUs Page and have a "host" compute node linked.

Most actions you can take on compute nodes can also be taken on DPUs. For example, you can deploy images and PXE boot DPUs managed by the ClusterWareAI software. Soft power control actions, such as reboot or shutdown, are taken directly on the DPU. Other power control actions, such as power on, power off, or power cycle, are taken on the host compute node associated with the DPU even if the action is run using the cw-dpuctl tool.

ClusterWareAI DPU support is available for NVIDIA BlueField-3 DPUs with DOCA. Additional support may be added in future releases. Contact Penguin Solutions to discuss alternative requirements.

Creating DPUs#

Use the cw-dpuctl tool to create a DPU. Specify a MAC address and a compute node host when creating a DPU. For example:

cw-dpuctl create mac=11:11:11:22:22:22 host=n0

DPUs, like compute nodes and switches, are automatically added to the DefaultAttribs attribute group and use the nX naming pool pattern. To differentiate DPUs from compute nodes and switches, consider using a different naming pool for DPUs. See Node Names and Pools for details and Switch Naming Pools for an example.

DPU and compute node host associations are available when viewing details for the host compute node as long as both the DPU and host compute node are up. Matched DPUs appear under dpus and under hardware.dpus. A parenthetical name, such as 1 (dpu2), is the linked DPU.

For example:

[admin@head0 ~]$ cw-nodectl -i n10 ls -l
Nodes
  n10
    dpus
      dpu2
    hardware
      dpus
        0
          pci_address: 0000:01:00
          pci_id: 15b3:c2d5
          product_serial: MT2623603M4F
        1 (dpu2)
          pci_address: 0004:01:00
          pci_id: 15b3:c2d5
          product_serial: MT2623603M4W

Booting DPUs#

Secure boot on DPUs is not supported. Disable secure boot before booting a DPU with the ClusterWareAI software.

Disable secure boot via BIOS or Redfish. For example:

curl -k -u name:password -X PATCH -H 'Content-Type: application/json' \
  "https://dpu-bmc0/redfish/v1/Systems/Bluefield/SecureBoot" \
  -d '{"SecureBootEnable": false}'

Reboot the DPU:
```
cw-dpuctl -i <DPU> reboot
```
The system may require multiple reboots before the change takes effect.

Verify the current state of the DPU by comparing the SecureBootCurrentBoot and SecureBootEnabled fields:

[admin@head0 ~]$ curl -k -u name:password "https://dpu-bmc0/redfish/v1/Systems/Bluefield/SecureBoot"
{
  "@odata.id": "/redfish/v1/Systems/Bluefield/SecureBoot",
  "@odata.type": "#SecureBoot.v1_1_0.SecureBoot",
  "Description": "The UEFI Secure Boot associated with this system.",
  "Id": "SecureBoot",
  "Name": "UEFI Secure Boot",
  "SecureBootCurrentBoot": "Enabled",
  "SecureBootDatabases": {
    "@odata.id": "/redfish/v1/Systems/Bluefield/SecureBoot/SecureBootDatabases"
  },
  "SecureBootEnable": false,
  "SecureBootMode": "UserMode"
}

When working with DPUs, you may need to adjust the kernel command line on the boot configuration to allow the DPU OS to start. For example:

cw-bootctl -i DpuBoot update cmdline="selinux=0 ro console=hvc0 console=ttyAMA0 \
   earlycon=pl011,0x13010000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1"

Updating the cmdline overwrites the existing cmdline, so make sure to include existing commands if needed.

DPU Images#

Like compute nodes, DPUs support deploying images via an associated boot configuration (DefaultBoot and DefaultImage by default). Alternatively, you can use Ignition or Kickstart to deploy DPU images.

Installing DOCA Runtime#

NVIDIA BlueField DPUs typically need the DOCA runtime stack, including Mellanox Firmware Tools (MFT) in the DPU image before you can use DPU devices or tools such as mlxprivhost. The ClusterWareAI software provides an example helper at /opt/scyld/clusterware-tools/examples/deploy/doca-runtime.sh. The example helper supports RHEL-compatible images that use dnf as well as Ubuntu 24.04 aarch64 images.

Apply the helper to an image using cw-modimg. For example:

cw-modimg -i Rocky-9.8-dpu-updated --deploy \
  /opt/scyld/clusterware-tools/examples/deploy/doca-runtime.sh

Restricting Compute Node Host Access#

You can restrict what the linked compute node host may do to the DPU card using mlxprivhost. For example, limit firmware flashing via the host. If MFT is not yet present in the image, see Installing DOCA Runtime.

Prerequisite: Start the Mellanox Software Tools (MST). See https://networking-docs.nvidia.com/mftswum/4.36.0/linux for details.

Confirm a device is present:

mst start
mst status

Example output:

MST devices:
------------
/dev/mst/mt41692_pciconf0        - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:03:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 01

Query the current host privilege level:

mlxprivhost -d /dev/mst/mt41692_pciconf0 q

Example (privileged host):

Host configurations
-------------------
level                         : PRIVILEGED

Port functions status:
-----------------------
disable_rshim                 : FALSE
disable_tracer                : FALSE
disable_port_owner            : FALSE
disable_counter_rd            : FALSE

Restrict the host with the r action. Optional flags, such as --disable_rshim, further limit host functions. For example:
```
mlxprivhost -d /dev/mst/mt41692_pciconf0 r --disable_rshim
```
If you changed the --disable_rshim setting, power cycle the host and DPU.

Confirm the restrictions applied as expected. In the example, level should be RESTRICTED and any disabled functions set to TRUE:

mlxprivhost -d /dev/mst/mt41692_pciconf0 q
Host configurations
-------------------
level                         : RESTRICTED

Port functions status:
-----------------------
disable_rshim                 : TRUE
disable_tracer                : FALSE
disable_port_owner            : FALSE
disable_counter_rd            : FALSE

To restore a privileged host, use the p action:

mlxprivhost -d /dev/mst/mt41692_pciconf0 p
-I- preparing configuration file...Done!
-I- configuring device...Done!

For flag details and zero-trust behavior, see https://networking-docs.nvidia.com/mftswum/4350/mlxprivhost-nic-configuration-by-the-host-restriction-tool-zero-trust-mode.