Deploying Images Using Ignition#

To partition nodes during their boot process, the ClusterWareAI ™ platform leverages tools called Ignition and Butane. Use Ignition and Butane for persistent boot nodes, such as login nodes, storage servers, or other infrastructure nodes.

Both Ignition and Butane originate from the Red Hat CoreOS project. The Ignition tool accepts a machine parsable JSON-formatted configuration file while the Butane tool translates a human-readable YAML format and produces the JSON file that Ignition needs. Details of these projects can be found at:

Butane: https://coreos.github.io/butane/
Ignition: https://coreos.github.io/ignition/

The functionality provided by Ignition enables the ClusterWareAI platform to partition and format local disks and deploy a root file system image to the newly prepared drives. This bypasses the previously used kickstart process to deploy locally-installed systems. The behavior of Ignition is controlled through a configuration file and node attributes. Another attribute triggers the installation of the GRUB2 bootloader, resulting in a fully provisioned and ready-to-boot system. By default these systems still use the ClusterWareAI platform for DHCP and report status back to the head nodes, but post-install changes can reconfigure the networking or disable the ClusterWareAI and Telegraf agents.

Most of the required node attributes are set in a pre-configured attribute group called IgnitionDeploy. The IgnitionDeploy attribute group uses the included partitions.butane file. The following table lists the attributes included in IgnitionDeploy attribute group by default. You can modify these values by cloning the IgnitionDeploy attribute group or by editing the group directly.

Attribute	Value	Description	Required
_boot_style	disked	Must be "disked"	yes
_disk_root	LABEL=root	Reference to the new root partition / file system	yes
_ignition	partitions.butane	Name of the Butane configuration file ending in .butane	yes
_bootloader	grub	Only the standard GRUB boot loader is currently supported.	yes
_wipe_all	now	Setting to "now" irreversibly wipes all data from local disks.	no

Note

One substantial difference between using Ignition directly and using it via the ClusterWareAI platform is that Ignition is intended to either produce exactly the desired result or nothing at all. However, when triggered by the _ignition attribute, the tool logs failures and proceeds. Future ClusterWareAI releases may change this behavior to more closely match the standard Ignition approach.

Ignition and Butane Configuration File Examples#

A simple Ignition file is available at /opt/scyld/clusterware/kickstarts/partitions.butane and is referenced by the IgnitionDeploy attribute group. The same file is also available in the /opt/scyld/clusterware-tools/examples/ directory.

The partitions.butane example file partitions the local drive (/dev/sda) into three partitions, /, /boot, and /boot/efi where /boot is 1024 MB in size, /boot/efi is 200 MB in size, and / encompasses the rest of the drive. A oneshot systemd unit is also created as part of the Butane file to reboot the system to the local installation at the end of deployment process.

A more advanced Butane file is available at /opt/scyld/clusterware-tools/examples/mdraids.butane and builds on the simple example. The mdraids.butane file demonstrates how to deploy a disk configuration that mirrors partitions on a pair of drives while using other drives as a raid5 scratch partition to prevent data loss. In the example, the mirror includes the /boot/efi partition. The file system is on md-root.

Tip

When starting with either example, review all of the partitions that are created to ensure their mount points exist within the image you will deploy.

Additional examples of Butane configuration files can be found in the Butane project documentation: https://coreos.github.io/butane/examples/

Provision Nodes Using Ignition#

To use Ignition during node provisioning:

Create an image that contains the desired operating system and software tools.

Note

Older ClusterWareAI images may not contain all the tools necessary for Ignition to properly partition or format the drives, such as sgdisk or mkfs.xfs. Ignition may rely on versions of tools that are not available on older operating systems.
Create a boot configuration using the image you just created:
```
cw-add-boot-config --image <IMAGE NAME> --boot-config <BOOT NAME>
```
Where <IMAGE NAME> is the name of the image you just created and <BOOT NAME> is a name for the new boot configuration.

Add the new boot configuration to the target node:

cw-nodectl -i <NODE NAME> set _boot_config=<BOOT NAME>

Add or update the Butane configuration file.
- If you are using the default /opt/scyld/clusterware/kickstarts/partitions.butane file, you may need to make changes to the file to match your configuration. For example, the device line references /dev/sda, but your drives may use a different name. For virtual machines, the drive is commonly /dev/vda. For bare metal, the drive is typically nvme0n1 or similar. On el9 and later kernels nvme drives can come up in a different order upon reboot. For these systems, identifying the drive using a path in /dev/disk/by-path/ is recommended for consistency across machines and boots.
- If you are using a custom configuration file, copy the file to the /opt/scyld/clusterware/kickstarts directory on all head nodes. As long as the file name ends with “.butane” the file is translated by the Butane command at download time. A file with any other extension must be a valid Ignition configuration file, as it is passed directly to the Ignition command without translation.
[Optional] If you are using a custom configuration file, use the _ignition reserved attribute to reference the custom file.

To update the file within the IgnitionDeploy attribute group:
```
cw-attribctl -i IgnitionDeploy set _ignition=<CONFIG FILE>
```
To set the attribute directly on the node:
```
cw-nodectl -i <NODE> set _ignition=<CONFIG FILE>
```
See Multiple Ignition Configuration Files for additional options.
Add the target node to the IgnitionDeploy attribute group:
```
cw-nodectl -i <NODE NAME> join IgnitionDeploy
```
Reboot the node.

The node boots using the kernel and initramfs from the specified boot configuration. Within the initramfs, the system optionally wipes all data from the local disks and then executes the various Ignition stages during the normal image download and unpacking process as follows:
- After networking is established and the node has downloaded its initial attributes.ini file, it detects the _ignition attribute and downloads the Ignition binary from the head node.
- After downloading the root file system, the node runs the “fetch”, “kargs”, and “disks” stages to download the configuration, check the kernel arguments, and partition local disks.
- Immediately before unpacking the file system, the node runs the “mount” stage and then proceeds to unpack the image into the mounted partitions.
- After unpacking the image, the node runs the “files” stage to manipulate the unpacked file system.
Once Ignition completes, the _bootloader attribute triggers installation of the GRUB2 bootloader as a separate step. A successful installation of grub also updates _boot_style to ensure nodes do not continually loop through deployment. Finally, a reboot and relabel of the file system is triggered.

An ignition.log file is produced during the deployment process and can be found in /opt/scyld/clusterware-node/atboot on the booted node. If errors occur, it's likely that the node will not be able to boot. In that case, you can log in to the initramfs shell during the 20 second “press any key” countdown and examine the log file located at /atboot/ignition.log.

Multiple Ignition Configuration Files#

Since the Butane files are specific to disk, you may need additional versions of a Butane file if you are deploying to different hardware or to both bare metal and virtual machines. There are a few options when you have multiple Butane files:

Clone the IgnitionDeploy attribute group and have each attribute group reference a different Butane file.
Keep a single IgnitionDeploy attribute group and create other attribute groups that only set the _ignition reserved attribute. When joining nodes, make sure they join IgnitionDeploy before the new groups so that the _ignition reserved attribute is set properly (the joining order of attribute groups matter, see Attribute Inheritance).
Keep a single IgnitionDeploy attribute group and set the _ignition reserved attribute directly on applicable nodes.

Re-Image Nodes Provisioned Using Ignition#

If you need to re-image a node after setting up Ignition, set the node to boot from the ClusterWareAI software and re-image by repeating the same steps. For example:

Reset the _wipe_all and _boot_style reserved attributes.
Reboot the node to reprovision.

Troubleshoot Ignition Deployed Images#

If you get a timeout for the root device in your ignition.log file, make sure that the _wipe_all reserved attribute is set to "now". The Linux kernel may detect existing raid devices, bring them up, and cause Ignition to fail when it tries waiting for a newly created device. Setting _wipe_all=now removes all devices and allows Ignition to deploy on a fresh system.

When deploying to RAM, the node requires a reboot after an image update. If the node reboot fails or does not occur, check the bootloader.log file for issues.