Modifying Images

Once you have an existing image, you can install additional RPMs into that image. A recommended best practice is to rarely and only very carefully modify DefaultImage and DefaultBoot. Instead, use them as stable baselines from which you clone new images and boot configurations.

The scyld-modimg tool supports a rich collection of options. See scyld-modimg for details.

For example:

scyld-imgctl -i DefaultImage clone name=mpiImage
scyld-add-boot-config --image mpiImage --boot-config mpiBoot
scyld-modimg -i mpiImage --install openmpi3.1

Suppose you want to create a new boot config mpiAltBoot that references the same mpiImage, though is otherwise different than mpiBoot. For instance, if you want mpiAltBoot to have a different cmdline:

scyld-bootctl -i mpiBoot clone name=mpiAltBoot

# Note that an updated cmdline replaces the entire existing cmdline,
# so examine the current cmdline:
scyld-bootctl -i mpiAltBoot ls -l | grep cmdline
# and perhaps the current cmdline is "enforcing=0", which you add to a new cmdline:
scyld-bootctl -i mpiAltBoot update cmdline="enforcing=0 console=ttyS1,115200"

You can manually customize an image, including installing or removing RPMs and modifying configuration files, by operating on the image inside a chroot:

scyld-modimg -i mpiImage --chroot

You can also combine commands, ending inside a chroot:

scyld-modimg --create --set-name mpiImage --install openmpi3.1 --chroot

If scyld-modimg --chroot detects a problem accessing or manipulating the local image, delete the local image (see Deleting Unused Images), and then the retry of the operation will download a fresh copy of the remote image into the cache. Alternatively, execute scyld-modimg and add the --freshen argument, which ignores the current cached local image and downloads a fresh copy.

Inside the chroot, execute as user root and manually add, update, or remove rpms with yum (or other appropriate package manager), modify configuration files, etc. When you exit the chroot, you are asked if you want to discard or keep the changes. If you keep the changes, then you are asked whether or not you want to replace the local image, to upload the local image, and to replace the remote image.

Note

Keep in mind that several directories in the image do not get repacked and saved into the image file after an exit. Among them are /tmp/, /var/tmp/, and /var/cache/yum.

If your intention is to answer yes to all the questions following your exit, then you can skip those questions by adding more arguments to the original command line:

scyld-modimg --create --set-name mpiImage --install openmpi3.1 --chroot \
                      --no-discard --overwrite --upload

Examine the RPM contents of an image without going into a chroot by doing a simple query:

# Display the version of 'clusterware-node' in the image
scyld-modimg -i mpiImage --query clusterware-node

# Display the version of all RPMs in the image
scyld-modimg -i mpiImage --query

Finally, you must set the _boot_config attribute for specific nodes, or for all nodes, as desired to use this new boot config. For example, to have nodes n0-n15 use the mpiBoot boot config:

scyld-nodectl -i n[0-15] set _boot_config=mpiBoot

The scyld-modimg command prompts you about whether to overwrite an existing image or create a new one. It also prompts about whether to upload the resulting file to the head node, optionally overwriting the image stored on the ICE ClusterWare™ head node. This tool operates on a local cache of the image and cannot be used to delete an image from the head nodes or to directly modify the name or description of an image on the head node. To modify these sorts of fields, use the scyld-imgctl tool.

Images are stored in the head node's /opt/scyld/clusterware/storage/ directory in cwsquash format, which consists of a squashfs image offset inside a pseudo-disk image. This format is suitable for exporting via iSCSI.

Small homogeneous clusters may use a single node image across all compute nodes, although larger clusters that include compute nodes with differing hardware will require additional customization that may not be applicable to all nodes. Although you might find that node attributes (discussed in more detail in Interacting with Compute Nodes) and customized boot-time scripting provide adequate image customization, it may be useful (or necessary) to create additional boot configurations and root file systems that meet specific hardware and/or software needs.

Customization can involve more than adding software drivers to support node-specific hardware and adding applications and their associated software stacks. It can also involve customizing configuration files in an image to deal with a non-standard networking environment. For example, if the compute node needs to use a networking route that is not the gateway defined in the head node's /opt/scyld/clusterware-iscdhcp/dhcpd.conf.template, then you can edit that file to modify the default option routers <GATEWAY>; line, or edit the compute node image's appropriate /etc/sysconfig/network-scripts/ifcfg-* script to insert the desired GATEWAY IP address. For more details see the documentation for your base distribution.