Modifying Images#

Once you have an existing image, you can install additional RPMs into that image. A recommended best practice is to rarely and only very carefully modify DefaultImage and DefaultBoot. Instead, use them as stable baselines from which you clone new images and boot configurations.

The cw-modimg tool supports a rich collection of options. For example:

cw-imgctl -i DefaultImage clone name=mpiImage
cw-add-boot-config --image mpiImage --boot-config mpiBoot
cw-modimg -i mpiImage --install openmpi3.1

Tip

You can provide a boot configuration or node as input to the cw-modimg command and it will identify and modify the image associated with the boot configuration or node.

Suppose you want to create a new boot config mpiAltBoot that references the same mpiImage, though is otherwise different than mpiBoot. For instance, if you want mpiAltBoot to have a different cmdline:

cw-bootctl -i mpiBoot clone name=mpiAltBoot

# Note that an updated cmdline replaces the entire existing cmdline,
# so examine the current cmdline:
cw-bootctl -i mpiAltBoot ls -l | grep cmdline
# and perhaps the current cmdline is "enforcing=0", which you add to a new cmdline:
cw-bootctl -i mpiAltBoot update cmdline="enforcing=0 console=ttyS1,115200"

You can manually customize an image, including installing or removing RPMs and modifying configuration files, by operating on the image inside a chroot:

cw-modimg -i mpiImage --chroot

You can also combine commands, ending inside a chroot:

cw-modimg --create --set-name mpiImage --install openmpi3.1 --chroot

If cw-modimg --chroot detects a problem accessing or manipulating the local image, delete the local image (see Deleting Unused Images), and then the retry of the operation will download a fresh copy of the remote image into the cache. Alternatively, execute cw-modimg and add the --freshen argument, which ignores the current cached local image and downloads a fresh copy.

Inside the chroot, execute as user root and manually add, update, or remove rpms with yum (or other appropriate package manager), modify configuration files, etc. When you exit the chroot, you are asked if you want to discard or keep the changes. If you keep the changes, then you are asked whether or not you want to replace the local image, to upload the local image, and to replace the remote image.

Note

Keep in mind that several directories in the image do not get repacked and saved into the image file after an exit. Among them are /tmp/, /var/tmp/, and /var/cache/yum.

If your intention is to answer yes to all the questions following your exit, then you can skip those questions by adding more arguments to the original command line:

cw-modimg --create --set-name mpiImage --install openmpi3.1 --chroot \
                      --no-discard --overwrite --upload

Examine the RPM contents of an image without going into a chroot by doing a simple query:

# Display the version of 'clusterware-node' in the image
cw-modimg -i mpiImage --query clusterware-node

# Display the version of all RPMs in the image associated with node n1
cw-modimg -i n1 --query

Finally, you must set the _boot_config attribute for specific nodes, or for all nodes, as desired to use this new boot config. For example, to have nodes n0-n15 use the mpiBoot boot config:

cw-nodectl -i n[0-15] set _boot_config=mpiBoot

The cw-modimg command prompts you about whether to overwrite an existing image or create a new one. It also prompts about whether to upload the resulting file to the head node, optionally overwriting the image stored on the ICE ClusterWare ™ head node. This tool operates on a local cache of the image and cannot be used to delete an image from the head nodes or to directly modify the name or description of an image on the head node. To modify these sorts of fields, use the cw-imgctl tool.

Tip

If an image is missing the localhost entries in the /etc/hosts file, you can re-add the entries by:

  1. Running cw-modimg --chroot into the image.

  2. Re-adding the localhost entries to the /etc/hosts files.

  3. Setting the /etc/hosts file permissions to 644.

  4. Exiting and saving the image.

Images are stored in the head node's /opt/scyld/clusterware/storage/ directory in cwsquash format, which consists of a squashfs image offset inside a pseudo-disk image. This format is suitable for exporting via iSCSI.

Small homogeneous clusters may use a single node image across all compute nodes, although larger clusters that include compute nodes with differing hardware will require additional customization that may not be applicable to all nodes. Although you might find that node attributes (discussed in more detail in Interacting with Compute Nodes) and customized boot-time scripting provide adequate image customization, it may be useful (or necessary) to create additional boot configurations and root file systems that meet specific hardware and/or software needs.

Customization can involve more than adding software drivers to support node-specific hardware and adding applications and their associated software stacks. It can also involve customizing configuration files in an image to deal with a non-standard networking environment. For example, if the compute node needs to use a networking route that is not the gateway defined in the head node's /opt/scyld/clusterware-iscdhcp/dhcpd.conf.template, then you can edit that file to modify the default option routers <GATEWAY>; line, or edit the compute node image's appropriate /etc/sysconfig/network-scripts/ifcfg-* script to insert the desired GATEWAY IP address. For more details see the documentation for your base distribution.