Using Kubernetes#

This section provides examples of setting up Kubernetes clusters with ICE ClusterWare™ and non-ClusterWare systems.

See Kubernetes for a general explanation of how to install and initialize a Kubernetes cluster and scyld-kube for details about the command and related arguments.

Note

All examples assume you have root user or ClusterWare administrator access and that the clusterware-kubeadm package is installed.

Using a Single Non-ClusterWare System as a Control Plane#

On the non-ClusterWare system where clusterware-kubeadm is installed, kube1 (10.154.3.1), initialize the local system as a control plane:

scyld-kube --init

The following messages are printed out after the control plane initialization:

...
To join ClusterWare NODES/IMAGE as worker to this non ClusterWare control plane:
scyld-kube -i NODES --join --token nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1
scyld-kube --image IMAGE --join --token nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1

To join non ClusterWare system as worker to this non Clusterware control plane:
scyld-kube --join --token nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1
...

Verify the deployment status by running the following command:
```
kubectl get nodes -o wide
```
Using the messages at the end of step 1 as a guide, join ClusterWare nodes (n[11-14]) as workers with explicit --token, --cahash, and --cluster arguments to the control plane kube1 (10.154.3.1):
```
scyld-kube -i n[11-14] --join nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1
```

Create a Kubernetes worker node image with explicit --token, --cahash, and --cluster arguments then boot n[15-20] with the node image as workers to control plane kube1 (10.154.3.1):

$ scyld-bootctl -i DefaultBoot clone name=KubeWorkerBoot2
$ scyld-imgctl -i DefaultImage clone name=KubeWorkerImage2
$ scyld-kube --image KubeWorkerImage2 --join --token nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1
$ scyld-bootctl -i KubeWorkerBoot2 up image=KubeWorkerImage2
$ scyld-nodectl -i n[15-20] set _boot_config=KubeWorkerBoot2
$ scyld-nodectl -i n[15-20] reboot

On EACH non-ClusterWare system that you want to join as a worker and where clusterware-kubeadm is installed, join the local system to control plane kube1 (10.154.3.1) with explicit --token, --cahash, and --cluster arguments:
```
scyld-kube --join --token nfg0ku.73f1gre8gxzco1qx --cahash sha256:cc999d4001c018a3423238773614bb8d6d8ad720e1f31a8b0e862052a67262da --cluster 10.154.3.1
```
Verify the deployment status by running the following command:
```
kubectl get nodes -o wide
```
You should see kube1 as the control plane and both the ClusterWare and non-ClusterWare systems you joined as workers in the output.

Using Multiple ClusterWare Nodes as a Control Plane#

Create High Available (HAProxy and Keepalived) configure files with ClusterWare node n21 (10.154.1.121) as the first control plane node and n22 (10.154.1.122) and n23 (10.154.1.123) as additional control plane nodes:
```
scyld-kube --prepare-lb 10.154.2.0 n21:10.154.1.121,n22:10.154.1.122,n23:10.154.1.123
```
Note

10.154.2.0 is an unused IP within the cluster network. It will be the apiserver virtual IP for these Kubernetes control planes.

Initialize the first control plane node on n21:

scyld-kube -i n21 --init-ha

The following message is printed out from a successful initialization:

...
To join ClusterWare NODES as control planes to this ClusterWare control plane:
scyld-kube -i NODES --join-ha --certificate-key 1271738c2ee3cda4dc022a9bef8a3166550a608e80d000cdf0dfbe3defb03776 --cluster n21
...

Note

There will also be messages about joining non-ClusterWare systems as workers to this ClusterWare control plane.

Verify the first control plane node is ready and note the --cluster value with INTERNAL-IP. See Checking Deployment Status. If it is more than 2 hours since the first control plane node was initialized, generate a new certificate key. See Additional Configuration.

Join n22 and n23 as additional control plane nodes to the first control plane node (n21):

scyld-kube -i n[22-23] --join-ha --certificate-key 1271738c2ee3cda4dc022a9bef8a3166550a608e80d000cdf0dfbe3defb03776 --cluster n21

Verify all control plane nodes are ready. See Checking Deployment Status.
Using the messages at the end of step 2 as a guide, join ClusterWare nodes (n[1-4]) as workers to the control plane node n21:
```
scyld-kube -i n[1-4] --join --cluster n21
```

Create a Kubernetes worker node image and then boot n[5-10] with the node image as workers to the control plane node n21:

$ scyld-bootctl -i DefaultBoot clone name=KubeWorkerBoot
$ scyld-imgctl -i DefaultImage clone name=KubeWorkerImage
$ scyld-kube --image KubeWorkerImage --join --cluster n21
$ scyld-bootctl -i KubeWorkerBoot up image=KubeWorkerImage
$ scyld-nodectl -i n[5-10] set _boot_config=KubeWorkerBoot
$ scyld-nodectl -i n[5-10] reboot

On EACH non-ClusterWare system that you want to join as a worker and where clusterware-kubeadm is installed, join the local system to the control plane node n21 with explicit --token, --cahash, and --cluster arguments:
```
scyld-kube --join --token yp6lxa.wcb6g48ud3f2cwng --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --cluster 10.154.2.0
```

Using Multiple Non-ClusterWare Systems as a Control Plane#

On EACH non-ClusterWare system where clusterware-kubeadm is installed, create High Available (HAProxy and Keepalived) configure files with kube2 (10.154.3.2) as the first control plane node and kube3 (10.154.3.3) and kube4 (10.154.3.4) as additional control plane nodes:
```
scyld-kube --prepare-lb 10.154.4.0 kube2:10.154.3.2,kube3:10.154.3.3,kube4:10.154.3.4
```
Note

10.154.4.0 is an unused IP within the cluster network. It will be the apiserver virtual IP for these Kubernetes control planes.

Initialize the control plane on kube2:

scyld-kube --init-ha

The following message is printed out from a successful initialization:

...
To join non ClusterWare system as control plane to this non ClusterWare control plane:
scyld-kube --join-ha --token ka8y8y.enwcyfsk4hblayz5 --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --certificate-key 86ae5340eb592759debd51ab9a03c9f9005a5027e7900d3a2fff687de473e2be --cluster 10.154.4.0
...

Note

There will also be messages about joining ClusterWare NODES/IMAGE as workers to this non-ClusterWare control plane.

On kube2, verify the first control plane node is ready. See Checking Deployment Status. If it is more than 2 hours since the first control plane node was initialized, generate a new certificate key. See Additional Configuration.

On kube3, create the same High Available (HAProxy and Keepalived) configure files as on kube2 and then join kube3 as an additional control plane node:

$ scyld-kube --prepare-lb 10.154.4.0 kube2:10.154.3.2,kube3:10.154.3.3,kube4:10.154.3.4
$ scyld-kube --join-ha --token ka8y8y.enwcyfsk4hblayz5 --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --certificate-key 86ae5340eb592759debd51ab9a03c9f9005a5027e7900d3a2fff687de473e2be --cluster 10.154.4.0

Repeat step 4 on kube4.
Verify all control planes nodes are ready. See Checking Deployment Status.
Using the messages at the end of step 2 as a guide, join ClusterWare nodes (n[11-14]) as workers with explicit --token, --cahash, and --cluster arguments to the control plane node kube2 (10.154.4.0):
```
scyld-kube -i n[11-14] --join --token ka8y8y.enwcyfsk4hblayz5 --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --cluster 10.154.4.0
```

Create a Kubernetes worker node image with explicit --token, --cahash, and --cluster arguments and then boot n[15-20] with the node image as workers to the control plane node kube2 (10.154.4.0):

$ scyld-bootctl -i DefaultBoot clone name=KubeWorkerBoot2
$ scyld-imgctl -i DefaultImage clone name=KubeWorkerImage2
$ scyld-kube --image KubeWorkerImage2 --join --token ka8y8y.enwcyfsk4hblayz5 --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --cluster 10.154.4.0
$ scyld-bootctl -i KubeWorkerBoot2 up image=KubeWorkerImage2
$ scyld-nodectl -i n[15-20] set _boot_config=KubeWorkerBoot2
$ scyld-nodectl -i n[15-20] reboot

On EACH non-ClusterWare system that you want to join as a worker and where clusterware-kubeadm is installed, join the local system to the control plane node kube2 (10.154.4.0) with explicit --token, --cahash, and --cluster arguments:
```
scyld-kube --join --token ka8y8y.enwcyfsk4hblayz5 --cahash sha256:413a6267bac67ff749734749dc8b5f60323a68c64bf7fc8e99292dd9b29040b2 --cluster 10.154.4.0
```