Executing Commands

A cluster administrator can execute commands on one or more compute nodes using the cw-nodectl tool. For example:

cw-nodectl -i n0 exec ls -l /

passes the command, e.g. ls -l /, to the head node, together with a list of target compute nodes. The head node will then ssh to each compute node using the head node's SSH key, execute the command, and return the output to the calling tool that will display the results. Note that this relay through the REST API is done because the ICE ClusterWare ™ tools may be installed on a machine that is not a head node and is not able to directly access the compute nodes.

Note that even if DNS resolution of compute node names is not possible on the local machine, cw-nodectl exec will still work because it retrieves the node IP addresses from the ClusterWare database via the head node. Further, once an administrator has appropriate keys on the compute nodes and has DNS resolution of compute node names, they are encouraged to manage nodes either directly using the ssh or pdsh commands or at a higher level with a tool such as ansible.

Commands executed through cw-nodectl exec are executed in parallel across the selected nodes. By default 64 nodes are accessed at a time, but this is adjustable by setting the ssh_runner.fanout to a larger or smaller number. This variable can be set in an administrator's ~/.scyldcw/settings.ini or can be set in /opt/scyld/clusterware/conf/base.ini on a head node. Setting the ssh_runner.fanout variable to a value less than or equal to 1 causes all commands to be executed serially across the nodes.

Some limited support is also provided for sending content to the stdin of the remote command. That content can be provided in a file via an option, e.g.:

cw-nodectl -i n0 exec --stdin=@input.txt dd of=/root/output.txt

or the content can be provided directly:

cw-nodectl -i n0 exec --stdin='Hello World' dd of=/root/output.txt

or the content can be piped to cw-nodectl, and this time optionally using redirection on the compute node to write to the output file:

echo 'Hello world' | cw-nodectl -i n0 exec cat > /root/output.txt

When a command is executed on a single node, the command's stdout and stderr streams will be sent unmodified to the matching file descriptor of the cw-nodectl command. This allows an administrator to include remote commands in a pipe much like ssh. For example:

echo 'Hello world' | cw-nodectl -i n0 exec tr 'a-z' 'A-Z' > output.txt

will result in a the local file output.txt containing the text "HELLO WORLD". The cw-nodectl exec exit code will also be set to the exit code of the underlying command. When a command is executed on multiple nodes, the individual lines of the resulting output will be prefixed with the node names:

[admin@virthead]$ cw-nodectl -in[0-1] exec ls -l
n0: total 4
n0: -rw-r--r--. 1 root root 13 Apr  5 20:39 output.txt
n1: total 0

When executing a command on multiple nodes, the exit code of the cw-nodectl exec command will only be 0 if the command exits with a 0 on each node. Otherwise the tool return code will match the non-zero status of the underlying command from one of the failing instances.

The mechanism for passing stdin should not be used to transfer large amounts of data to the compute nodes, as the contents will be forwarded to the head node, briefly cached, and copied to all compute nodes. Further, if the data was passed as a stream either through piping to the cw-nodectl command or passing the path to a large file via the --stdin=@/path/to/file mechanism, the nodes will be accessed serially, not in parallel, so that the stream can be rewound between executions. This is supported for convenience when passing small payloads, but is not efficient in large clusters. A more direct method such as scp or pdcp should be used when the content is more than a few megabytes in size. Also note that even when communicating with a single compute node, this is not truly interactive because all of stdin must be available and sent to the head node before the remote command is executed.