Modifying Images#

Once you have an existing image, you can make changes to the image contents, such as installing additional RPMs into that image. A recommended best practice is to rarely and only very carefully modify DefaultImage and DefaultBoot. Instead, use them as stable baselines from which you clone new images and boot configurations.

The cw-modimg tool operates on a local cache of an image and supports a rich collection of options for modifying existing images. Some changes, such as deleting an image from the head nodes or modifying the name or description of an image on the head node, are done using the cw-imgctl tool. There are times when you would use both tools together. For example, you can use cw-imgctl to clone DefaultImage, use cw-add-boot-config to create a new boot config that references the new image, and finally add a new package to the new image:

cw-imgctl -i DefaultImage clone name=mpiImage
cw-add-boot-config --image mpiImage --boot-config mpiBoot
cw-modimg -i mpiImage --install openmpi3.1

Tip

You can provide a boot configuration or node as input to the cw-modimg command and it will identify and modify the image associated with the boot configuration or node.

Images are stored in the head node's /opt/scyld/clusterware/storage/ directory in cwsquash format, which consists of a squashfs image offset inside a pseudo-disk image. This format is suitable for exporting via iSCSI.

Small homogeneous clusters may use a single node image across all compute nodes, although larger clusters that include compute nodes with differing hardware will require additional customization that may not be applicable to all nodes. Although you might find that node attributes and customized boot-time scripting provide adequate image customization, it may be useful (or necessary) to create additional boot configurations and root file systems that meet specific hardware and/or software needs.

Image customization can involve more than adding software drivers to support node-specific hardware and adding applications and their associated software stacks. It can also involve customizing configuration files in an image to deal with a non-standard networking environment. For example, if the compute node needs to use a networking route that is not the gateway defined in the head node's /opt/scyld/clusterware-iscdhcp/dhcpd.conf.template, then you can edit that file to modify the default option routers <GATEWAY>; line, or edit the compute node image's appropriate /etc/sysconfig/network-scripts/ifcfg-* script to insert the desired GATEWAY IP address. See the documentation for your base distribution for details.

Modifying Images and Boot Configurations#

Suppose you want to create a new boot config mpiAltBoot that references the same mpiImage as an existing boot configuration mpiBoot, though is otherwise different than mpiBoot. For instance, if you want mpiAltBoot to have a different cmdline:

  1. Clone the existing boot configuration:

    cw-bootctl -i mpiBoot clone name=mpiAltBoot

  2. Updating the cmdline replaces the entire existing cmdline, so examine the current cmdline for the boot configuration:

    cw-bootctl -i mpiAltBoot ls -l | grep cmdline

  3. Add a new cmdline to the existing cmdline. In this example, the existing cmdline was "enforcing=0":

    cw-bootctl -i mpiAltBoot update cmdline="enforcing=0 console=ttyS1,115200"

  4. Set the _boot_config attribute on the nodes that you want to use the new boot configuration. For example, to have nodes n0-n15 use the mpiAltBoot boot config:

    cw-nodectl -i n[0-15] set _boot_config=mpiAltBoot

Modify Image Contents#

You can examine the RPM contents of an image with a simple query. For example, to display the version of clusterware-node in the image:

cw-modimg -i mpiImage --query clusterware-node

To display the version of all RPMs in the image associated with node n1:

cw-modimg -i n1 --query

You can manually customize an image, including installing or removing RPMs and modifying configuration files, by operating on the image inside a chroot or by mounting the image. The cw-modimg --chroot command unpacks the image to a randomly generated path in the ~/.scyldcw/workspace directory. By contrast, the cw-modimg --mount command downloads the image into the same directory and unpacks the image to the specified mounting location. After you finish making changes to the image, either exit the chroot or run the cw-modimg --unmount command.

The cw-modimg tool prompts you about whether to overwrite an existing image or create a new one. It also prompts about whether to upload the resulting file to the head node, optionally overwriting the image stored on the head nodes.

The cw-modimg tool then:

  1. Corrects the SELinux labels on files within the unpacked directory.

  2. Packs the image into a file in the workspace.

  3. Computes a checksum for the image.

  4. Deletes the unpacked version of the image.

  5. Uploads and stores the image in /opt/scyld/clusterware/storage.

For example, to modify an image with chroot:

cw-modimg -i mpiImage --chroot

You can also combine commands, ending inside a chroot:

cw-modimg --create --set-name mpiImage --install openmpi3.1 --chroot

If cw-modimg --chroot detects a problem accessing or manipulating the local image, delete the local image (see Deleting Unused Images), and then the retry of the operation downloads a fresh copy of the remote image into the cache. Alternatively, execute cw-modimg and add the --freshen argument, which ignores the current cached local image and downloads a fresh copy.

Inside the chroot, you can manually add, update, or remove rpms with dnf (or other appropriate package manager), modify configuration files, and so on. When you exit the chroot, you are asked if you want to discard or keep the changes. If you keep the changes, then you are asked whether or not you want to replace the local image, to upload the local image, and to replace the remote image.

Note

Keep in mind that several directories in the image are not repacked and saved into the image file after an exit. Among them are /tmp/, /var/tmp/, and /var/cache/yum.

If your intention is to answer yes to all the questions following your exit, then you can skip those questions by adding more arguments to the original command. For example:

cw-modimg --create --set-name mpiImage --install openmpi3.1 --chroot \
  --no-discard --overwrite --upload

Resolve Missing Localhost Entries#

If an image is missing the localhost entries in the /etc/hosts file, you can re-add the entries by:

  1. Running cw-modimg --chroot into the image.

  2. Re-adding the localhost entries to the /etc/hosts files.

  3. Setting the /etc/hosts file permissions to 644.

  4. Exiting and saving the image.

Resolving Image Modification Issues#

In rare cases, the cw-modimg tool fails when packing and uploading the image changes made using chroot or --mount. Depending on when the failure occurs, you can often resume the process.

  • If failure occurs when correcting the SELinux labels, use the cw-modimg --unmount command to try again.

  • If failure occurs during packing or when computing a checksum, manually delete the corresponding ~/.scyldcw/workspace/<UID> file and run cw-modimg --unmount again.

  • If failure occurs when deleting the unpacked version of the image, run cw-modimg --upload to manually upload the image and manually clean up the unpacked image later.

  • If failure occurs when uploading the image, retry running cw-modimg --upload.

It is not common that you need to fully start over with image changes, though it may be simpler to do so if you are using automation.