Skip to main content

Question 20 - Update Kubernetes Version and Join Cluster

Question 20 | Update Kubernetes Version and join cluster​

Use context: kubectl config use-context k8s-c3-CCC

Your coworker said node cluster3-node2 is running an older Kubernetes version and is not even part of the cluster. Update Kubernetes on that node to the exact version that's running on cluster3-controlplane1. Then add this node to the cluster. Use kubeadm for this.

# Upgrading kubeadm clusters

# cluster3-node2 really isn't here, let's enter it and make it available at the same version as the cluster
k get node
NAME                     STATUS   ROLES           AGE     VERSION
cluster3-controlplane1   Ready    control-plane   3h28m   v1.29.0
cluster3-node1           Ready    <none>          3h23m   v1.29.0

ssh cluster3-node2
# checking kubectl version
root@cluster3-node2:~# kubectl version --short
#Client Version: v1.28.5
#Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
#The connection to the server localhost:8080 was refused - did you specify the right host or port?

# checking kubelet version
root@cluster3-node2:~# kubelet --version
#Kubernetes v1.28.5

# checking kubeadm version
root@cluster3-node2:~# kubeadm version
#kubeadm version: &version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.0",##GitCommit:"3f7a50f38688eb332e2a1b013678c6435d539ae6", GitTreeState:"clean", BuildDate:"2023-12-13T08:50:10Z", #GoVersion:"go1.21.5", Compiler:"gc", Platform:"linux/amd64"}
#Above we can see that kubeadm is already installed in the wanted version, so we don't need to install it. Hence we can run:

root@cluster3-node2:~# kubeadm upgrade node
#couldn't create a Kubernetes client from file "/etc/kubernetes/kubelet.conf": failed to load admin kubeconfig: open /etc/kubernetes/kubelet.conf: no such file or directory
#To see the stack trace of this error execute with --v=5 or higher
#This is usually the proper command to upgrade a node. But this error means that this node was never even initialised, so nothing to update here. This will be done later using kubeadm join. For now we can continue with kubelet and kubectl:

root@cluster3-node2:~# apt update
Hit:1 http://ppa.launchpad.net/rmescandon/yq/ubuntu focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease  
Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease  
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb  InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb  InRelease
Get:6 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:7 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:8 http://us.archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [919 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,023 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,141 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB]
Fetched 5,959 kB in 3s (2,049 kB/s)  
Reading package lists... Done
Building dependency tree  
Reading state information... Done

222 packages can be upgraded. Run 'apt list --upgradable' to see them.

➜ root@cluster3-node2:~# apt show kubectl -a | grep 1.29

Version: 1.29.0-1.1

APT-Sources: https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages

➜ root@cluster3-node2:~# apt install kubectl=1.29.0-1.1 kubelet=1.29.0-1.1

Reading package lists... Done
Building dependency tree  
Reading state information... Done
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 220 not upgraded.
Need to get 30.3 MB of archives.
After this operation, 782 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb  kubectl 1.29.0-1.1 [10.5 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb  kubelet 1.29.0-1.1 [19.8 MB]
Fetched 30.3 MB in 1s (40.8 MB/s)
(Reading database ... 112588 files and directories currently installed.)
Preparing to unpack .../kubectl_1.29.0-1.1_amd64.deb ...
Unpacking kubectl (1.29.0-1.1) over (1.28.5-1.1) ...
Preparing to unpack .../kubelet_1.29.0-1.1_amd64.deb ...
Unpacking kubelet (1.29.0-1.1) over (1.28.5-1.1) ...
Setting up kubectl (1.29.0-1.1) ...
Setting up kubelet (1.29.0-1.1) ...

➜ root@cluster3-node2:~# kubelet --version

Kubernetes v1.29.0
Now we're up to date with kubeadm, kubectl and kubelet. Restart the kubelet:

root@cluster3-node2:~# service kubelet restart

root@cluster3-node2:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2024-01-04 14:01:11 UTC; 2s ago
       Docs: https://kubernetes.io/docs/
    Process: 43818 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAIL>
   Main PID: 43818 (code=exited, status=1/FAILURE)
Jan 04 14:01:11 cluster3-node2 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jan 04 14:01:11 cluster3-node2 systemd[1]: kubelet.service: Failed with result 'exit-code'.
These errors occur because we still need to run kubeadm join to join the node into the cluster. Let's do this in the next step.
Add cluster3-node2 to cluster

First we log into the controlplane1 and generate a new TLS bootstrap token, also printing out the join command:

ssh cluster3-controlplane1
root@cluster3-controlplane1:~# kubeadm token create --print-join-command

kubeadm join 192.168.100.31:6443 --token pbuqzw.83kz9uju8talblrl --discovery-token-ca-cert-hash sha256:eae975465f73f316f322bcdd5eb6a5a53f08662ecb407586561cdc06f74bf7b2

root@cluster3-controlplane1:~# kubeadm token list
TOKEN                     TTL         EXPIRES                ...
dm3ws5.hga8xkwpp0f2lk4q   20h         2024-01-05T10:28:19Z
pbuqzw.83kz9uju8talblrl   23h         2024-01-05T14:01:38Z
rhjon6.qra3to1sjf2xnn0l   <forever>   <never>

We see the expiration of 23h for our token, we could adjust this by passing the ttl argument.

Next we connect again to cluster3-node2 and simply execute the join command:

➜ ssh cluster3-node2
root@cluster3-node2:~# kubeadm join 192.168.100.31:6443 --token pbuqzw.83kz9uju8talblrl --discovery-token-ca-cert-hash sha256:eae975465f73f316f322bcdd5eb6a5a53f08662ecb407586561cdc06f74bf7b2

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

➜ root@cluster3-node2:~# service kubelet status

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Thu 2024-01-04 14:02:45 UTC; 13s ago
       Docs: https://kubernetes.io/docs/
   Main PID: 44103 (kubelet)
      Tasks: 10 (limit: 462)
     Memory: 55.5M
     CGroup: /system.slice/kubelet.service
             └─44103 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/k>
If you have troubles with kubeadm join you might need to run kubeadm reset.
This looks great though for us. Finally we head back to the main terminal and check the node status:


k get node
NAME                     STATUS     ROLES           AGE     VERSION
cluster3-controlplane1   Ready      control-plane   3h34m   v1.29.0
cluster3-node1           Ready      <none>          3h29m   v1.29.0
cluster3-node2           NotReady   <none>          20s     v1.29.0

Give it a bit of time till the node is ready.

k get node
NAME                     STATUS   ROLES           AGE     VERSION
cluster3-controlplane1   Ready    control-plane   3h34m   v1.29.0
cluster3-node1           Ready    <none>          3h29m   v1.29.0
cluster3-node2           Ready    <none>          27s     v1.29.0