cw-dpuctl

NAME

cw-dpuctl -- Query and modify Data Processing Units (DPUs) for the cluster.

Note

This command is similar to cw-nodectl but only applies to DPUs.

USAGE

cw-dpuctl

[-h] [-v] [-q] [[-c | --config] CONFIG] [--base-url URL] [[-u | --user] USER[:PASSWD]] [--human | --json | --csv | --table] [--pretty | --no-pretty] [--fields FIELDS] [--show-uids] [-i DPUS] | -a] [--up | --down | --booting] [--selector TEXT] {list,ls, create,mk, clone,cp, update,up, delete,rm, get, set, clear, join, leave, status, hardware, reboot, shutdown, power, sol, ssh, script, exec, ping, waitfor, scp}

OPTIONAL ARGUMENTS

-h, --help

Print usage message and exit. Ignore trailing args, parse and ignore preceding args.

-v, --verbose

Increase verbosity.

-q, --quiet

Decrease verbosity.

-c, --config CONFIG

Specify a client configuration file CONFIG.

--show-uids

Do not try to make the output more human readable. Must be used with the list action.

-i, --ids DPUS

A comma-separated list of DPUs or an admin-defined group of DPUs to act upon. Values can include name, UID, or truncated UID.

-a, --all

Interact with all DPUs (default for list).

--up

Interact with all "up" DPUs.

--down

Interact with all "down" DPUs.

--booting

Interact with all "booting" DPUs.

--selector TEXT, -s TEXT

DPU selection string.

ARGUMENTS TO OVERRIDE BASIC CONFIGURATION DETAILS

--base-url URL

Specify the base URL of the ClusterWareAI REST API.

-u, --user USER[:PASSWD]

Masquerade as user USER with optional colon-separated password PASSWD.

FORMATTING ARGUMENTS

--human

Format the output for readability (default).

--json

Format the output as JSON.

--csv

Format the output as CSV.

--table

Format the output as a table.

--pretty

Indent JSON or XML output, and substitute human readable output for other formats.

--no-pretty

Opposite of --pretty.

--fields FIELDS

Select individual fields in the result or error.

ACTIONS

clear [-a | --all | NAME ... ]

Delete attribute name(s) and their value(s).

-a, --all

Delete all attributes.

clone (cp) [--content JSON | INI_FILE] [NAME=VALUE ...]

Copy DPU with new NAME/VALUE identifier pairs.

--content JSON | INI_FILE

Overwrite fields in the cloned DPU.

create (mk) [--content JSON | INI_FILE ] [NAME=VALUE ...]

Add a DPU, commonly by specifying its MAC address (e.g., mac=MACaddr, that assigns the next available number and associated IP address) and host=<compute node>. Host is a required argument when creating a DPU and can be specified using a compute node name, UID, or other identifier.

--content JSON | INI_FILE

Load this content into the database as a DPU.

delete (rm)

Delete DPU(s).

exec [--grouped] [--in-order] [--label] [--stdin IN] [--binary] [--stdout OUT] [--stderr ERR] CMD

Execute the CMD (double-quotes are optional) on DPU(s). The cw-dpuctl exec command passes its current stdin, stdout, and stderr to the remote command, or uses the --stdin, --stdout, and/or --stderr arguments to override the default(s) with a file.

When run via an ssh command (e.g. ssh cwhead cw-dpuctl --up exec uptime), that stdin should be provided and closed with Ctrl-d, or ssh should be passed the -t argument to force tty allocation. Otherwise the command detects stdin is a pipe and wait for end-of-file.

Commands executed on multiple DPUs execute in parallel. The degree of fan-out can be controlled through the ssh_runner.fanout configuration variable in base.ini. Because these commands execute in parallel, their output may be interleaved or not in DPU index order. Override this with grouped or --in-order arguments.

For sshpass functionality, see _remote_pass in the Reserved Attributes section of the ClusterWareAI documentation.

--grouped

Results are locally buffered and printed grouped by DPU.

--in-order

Output is printed in DPU index order, implies --grouped.

--label

Force output labeling, even if a single DPU is selected.

--stdin IN

Provide @file or input string as stdin for the CMD.

--binary

Treat CMD output as binary data.

--stdout OUT

Provide a filename OUT for the CMD stdout output. Any {} in the filename gets translated to the DPU name.

--stderr ERR

Provide a filename ERR for the CMD stderr output. Any {} in the filename gets translated to the DPU name. An ERR value consisting of the string STDOUT will merge stderr into stdout.

get

Get attribute values.

hardware

Show the "hardware" information subset of cw-dpuctl ls -L.

join GROUP ...

Append GROUP(S) to the DPU attribute group lists.

leave [-a | --all | GROUP ...]

Remove GROUP(S) from the DPU attribute group lists.

-a, --all

Remove DPU(s) from all groups (other than the global default).

list (ls) [--long | --long-long]

Show information about DPUs.

-l, --long

Show a subset of all optional information for each DPU.

-L, --long-long

Show all optional information for each DPU.

ping [COUNT]

ping the specified DPU(s) with COUNT packets (default 1).

power {on | off | cycle | status | setnext BOOTDEV}

Display or control the host compute node power state through the plugin defined by the host compute node's power_uri, usually ipmi. The options on, off, cycle, and status correspond to ipmitool actions.

power {on|off|cycle} [--force] [--host NAME]

--force

Perform the power control action regardless of _no_boot.

--host NAME

Required. Power actions take place on the host compute node associated with the DPU. Provide the host compute node name, UID, or other identifier when taking power actions.

The option setnext specifies the boot device or method to use for the next DPU boot. BOOTDEV choices are none, pxe, disk, and bios.

power setnext BOOTDEV

Boot DPU device from: none, pxe, disk, bios

reboot [--force]

Soft reboot DPU(s) using SSH to run the shutdown command. Ignore the reboot if the DPU's _no_boot is set to true (or t, 1, yes, y) or if _busy is set to true (or t, 1, yes, y), unless an overriding --force argument is supplied.

--force

Override the DPU's _no_boot attribute value when set to 1.

scp

Copy files to or from DPU(s).

script SCRIPT

Execute the specified ClusterWareAI SCRIPT (distributed in the clusterware-node package) on the specified DPU(s). The script name list (or ls) displays names of the available scripts, which generally execute automatically at boot time to facilitate various DPU initializations and have limited usefulness for later execution by a cluster administrator. However, the scripts fetch_hosts (re-download the list of head nodes) and update_keys (update SSH keys) may be useful in rare circumstances for a booted DPU.

set [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Set attribute value(s).

--content JSON | INI_FILE

Import the NAME/VALUE pairs from the file into the DPU attributes.

shutdown [--force]

Soft shutdown DPU(s) using ssh to run the shutdown command.

--force

Perform the shutdown action regardless of _no_boot.

sol

Start a serial-over-lan (SOL) connection using the local ipmitool.

--enable ID

If SOL payload is disabled, then attempt to enable for ID and retry.

--steal

If an SOL session is currently active for that DPU, then deactivate that session and retry.

ssh [--pubkey FILE]

Create an SSH connection to the specified DPU as the user root. This is done using a local SSH key that is temporarily copied to the DPU through the head node and removed after the command completes. The user can provide their own public key, or one will be generated and stored in ~/.scyldcw/tempauth.key.

--pubkey FILE

Specify a file containing a public key to use for this connection.

status [--long] [--long-long] [--health | --ars | --no-ars] [--refresh] [--counts]

Show DPU status.

--health

Show status based on _health attribute.

--ars

Show status based on _ars_state attribute.

--no-ars

Opposite of --ars.

-l, --long

Show a subset of all optional information for each DPU.

-L, --long-long

Show all optional information for each DPU.

--refresh

Show basic DPU states, refreshing for any state change.

--counts

Include DPU counts.

update (up) [--content JSON | INI_FILE ] [ NAME=VALUE ] ...

Modify DPU fields with new values.

--content JSON | INI_FILE

Overwrite this content into the database for a DPU.

waitfor [Options] COND

Complete when one or more of the specified DPUs meet the condition COND, which is either an expression or a file name prefixed with an at symbol (@). If no DPUs are specified, then defaults to --all.

--failure COND

Also complete if the failure condition becomes true.

--timeout SECS

Complete after SECS seconds if condition(s) never become true.

--name NAME

Use the currently defined COND state known as NAME, or define a new COND and remember it as NAME.

--load

Save the state maps into the database.

--activate [SECS]

Activate a state map for SECS seconds (default=120)

--deactivate

Deactivate a state map.

--delete NAME

Delete an existing state map NAME.

--show [NAME]

Show a list of all state maps, or optionally just the details of one.

--stream

Stream back ongoing results instead of returning the first result and exiting.

--skip

Do not use or print the initial DPU states.

--one-per

Stream DPU state changes with one DPU per line.

EXAMPLES

cw-dpuctl list

List all DPU names.

cw-dpuctl status

Shows the basic state of each DPU.

cw-dpuctl -in7 status

Shows the basic state of DPU n7.

cw-dpuctl -i n7 ls -L

Shows full information available for DPU n7.

cw-dpuctl -i %groupx ls -l

Shows an expanded information available for each DPU joined to the admin-defined group groupx.

cw-dpuctl create mac=00:25:90:0C:D9:3C host=n001

Add a new DPU to the end of the current list of DPUs. The new DPU has a host of compute node n001.

cw-dpuctl create mac=00:25:90:0C:D9:3C host=n001 index=10

Add a new DPU beyond the end of the current list of DPUs as DPU n10.

cw-dpuctl -i n3 update mac=40:25:88:0C:B9:2C

Replace the current MAC address for DPU n3 with a new MAC address.

cw-dpuctl -in2 ssh

Use ssh to open a shell on dpu n2.

cw-dpuctl -i n4 reboot then waitfor 's[state] == "up"'

Reboot dpu n4, then wait until the DPU returns to the "up" state.

cw-dpuctl -i n4 reboot then waitfor up

Reboot DPU n4, then wait until the DPU returns to the "up" state. Another supported shorthand is the conditional "down".

cw-dpuctl -i n0 ssh

Start a ssh session on DPU n0 as user root (by default) or whatever user is specified in the DPU's _remote_user attribute.

RETURN VALUES

Upon successful completion, cw-dpuctl returns 0. On failure, an error message is printed to stderr and cw-dpuctl returns 1.