Question 20 | Update Kubernetes
Use context: kubectl config use-context workload-stage
The cluster is running Kubernetes 1.29.5, update it to 1.30.1.
Use apt package manager and kubeadm for this.
Use ssh cluster3-controlplane1 and ssh cluster3-node1 to connect to the instances.
Answer:
Let's have a look at the current versions:
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster3-controlplane1 Ready control-plane 173m v1.29.5
cluster3-node1 Ready <none> 167m v1.29.5
Control Plane Components First we should update the control plane components running on the master node, so we drain it:
➜ k drain cluster3-controlplane1 --ignore-daemonsets
node/cluster3-controlplane1 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-nct5v, kube-system/weave-net-qfh9l
node/cluster3-controlplane1 drained
Next we ssh into it and check versions:
➜ ssh cluster3-controlplane1
➜ root@cluster3-controlplane1:~# kubelet --version
Kubernetes v1.29.5
➜ root@cluster3-controlplane1:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.1", GitCommit:"6911225c3f747e1cd9d109c305436d08b668f086", GitTreeState:"clean", BuildDate:"2024-05-14T10:49:05Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}
We see above that kubeadm is already installed in the required version. Else we would need to install it:
# not necessary because here kubeadm is already installed in correct version
apt-mark unhold kubeadm
apt-mark hold kubectl kubelet
apt install kubeadm=1.30.1-1.1
apt-mark hold kubeadm
Check what kubeadm has available as an upgrade plan:
➜ root@cluster3-controlplane1:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.29.5
[upgrade/versions] kubeadm version: v1.30.1
[upgrade/versions] Target version: v1.30.1
[upgrade/versions] Latest version in the v1.29 series: v1.29.5
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT NODE CURRENT TARGET
kubelet cluster3-controlplane1 v1.29.5 v1.30.1
kubelet cluster3-node1 v1.29.5 v1.30.1
Upgrade to the latest stable version:
COMPONENT NODE CURRENT TARGET
kube-apiserver cluster3-controlplane1 v1.29.5 v1.30.1
kube-controller-manager cluster3-controlplane1 v1.29.5 v1.30.1
kube-scheduler cluster3-controlplane1 v1.29.5 v1.30.1
kube-proxy 1.29.5 v1.30.1
CoreDNS v1.11.1 v1.11.1
etcd cluster3-controlplane1 3.5.12-0 3.5.12-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.30.1
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
And we apply to the required version:
➜ root@cluster3-controlplane1:~# kubeadm upgrade apply v1.30.1
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.30.1"
[upgrade/versions] Cluster version: v1.29.5
[upgrade/versions] kubeadm version: v1.30.1
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
...
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1488188576/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.1". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
Next we can check if our required version was installed correctly:
➜ root@cluster3-controlplane1:~# kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.30.1
[upgrade/versions] kubeadm version: v1.30.1
[upgrade/versions] Target version: v1.30.1
[upgrade/versions] Latest version in the v1.30 series: v1.30.1
Control Plane kubelet and kubectl
Now we have to upgrade kubelet and kubectl:
➜ root@cluster3-controlplane1:~# apt update
Hit:1 http://ppa.launchpad.net/rmescandon/yq/ubuntu focal InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb InRelease
Hit:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
➜ root@cluster3-controlplane1:~# apt install kubelet=1.30.1-1.1 kubectl=1.30.1-1.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.9 MB of archives.
After this operation, 10.2 MB disk space will be freed.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb kubectl 1.30.1-1.1 [10.8 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb kubelet 1.30.1-1.1 [18.1 MB]
Fetched 28.9 MB in 1s (36.8 MB/s)
(Reading database ... 112589 files and directories currently installed.)
Preparing to unpack .../kubectl_1.30.1-1.1_amd64.deb ...
Unpacking kubectl (1.30.1-1.1) over (1.29.5-1.1) ...
Preparing to unpack .../kubelet_1.30.1-1.1_amd64.deb ...
Unpacking kubelet (1.30.1-1.1) over (1.29.5-1.1) ...
Setting up kubectl (1.30.1-1.1) ...
Setting up kubelet (1.30.1-1.1) ...
➜ root@cluster3-controlplane1:~# apt-mark hold kubelet kubectl
kubelet set on hold.
kubectl set on hold.
➜ root@cluster3-controlplane1:~# service kubelet restart
➜ root@cluster3-controlplane1:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sun 2024-05-26 15:59:10 UTC; 3s ago
Docs: https://kubernetes.io/docs/
Main PID: 34143 (kubelet)
Tasks: 10 (limit: 1066)
Memory: 92.0M
CGroup: /system.slice/kubelet.service
└─34143 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/k>
...
➜ root@cluster3-controlplane1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
cluster3-controlplane1 Ready,SchedulingDisabled control-plane 3h6m v1.30.1
cluster3-node1 Ready <none> 179m v1.29.5
Done, only uncordon missing:
➜ k uncordon cluster3-controlplane1
node/cluster3-controlplane1 uncordoned
# Data Plane
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster3-controlplane1 Ready control-plane 3h6m v1.30.0
cluster3-node1 Ready <none> 3h v1.29.5
Our data plane consist of one single worker node, so let's update it. First thing is we should drain it:
➜ k drain cluster3-node1 --ignore-daemonsets
node/cluster3-node1 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-zlnnj, kube-system/weave-net-q82dz
evicting pod team-blue/pto-webform-557d7bc56b-w68zq
evicting pod kube-system/coredns-76f75df574-z5gsq
evicting pod kube-system/coredns-76f75df574-zzknh
evicting pod team-blue/pto-webform-557d7bc56b-rg8rt
evicting pod default/classification-bot-67644b99fd-jbq46
pod/pto-webform-557d7bc56b-w68zq evicted
pod/pto-webform-557d7bc56b-rg8rt evicted
pod/classification-bot-67644b99fd-jbq46 evicted
pod/coredns-76f75df574-zzknh evicted
pod/coredns-76f75df574-z5gsq evicted
node/cluster3-node1 drained
Next we ssh into it and upgrade kubeadm to the wanted version, or check if already done:
➜ ssh cluster3-node1
➜ root@cluster3-node1:~# apt update
Hit:1 http://ppa.launchpad.net/rmescandon/yq/ubuntu focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb InRelease
Hit:4 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.29/deb InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
➜ root@cluster3-node1:~# apt-mark unhold kubeadm
kubeadm was already not hold.
➜ root@cluster3-node1:~# apt-mark hold kubectl kubelet
kubectl set on hold.
kubelet set on hold.
➜ root@cluster3-node1:~# apt install kubeadm=1.30.1-1.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
kubeadm is already the newest version (1.30.1-1.1).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
➜ root@cluster3-node1:~# apt-mark hold kubeadm
kubeadm set on hold.
➜ root@cluster3-node1:~# kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2503851970/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
Now we follow that kubeadm told us in the last line and upgrade kubelet (and kubectl):
➜ root@cluster3-node1:~# apt-mark unhold kubectl kubelet
Canceled hold on kubectl.
Canceled hold on kubelet.
➜ root@cluster3-node1:~# apt install kubelet=1.30.1-1.1 kubectl=1.30.1-1.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.9 MB of archives.
After this operation, 10.2 MB disk space will be freed.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb kubectl 1.30.1-1.1 [10.8 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.30/deb kubelet 1.30.1-1.1 [18.1 MB]
Fetched 28.9 MB in 1s (43.0 MB/s)
(Reading database ... 112589 files and directories currently installed.)
Preparing to unpack .../kubectl_1.30.1-1.1_amd64.deb ...
Unpacking kubectl (1.30.1-1.1) over (1.29.5-1.1) ...
Preparing to unpack .../kubelet_1.30.1-1.1_amd64.deb ...
Unpacking kubelet (1.30.1-1.1) over (1.29.5-1.1) ...
Setting up kubectl (1.30.1-1.1) ...
Setting up kubelet (1.30.1-1.1) ...
➜ root@cluster3-node1:~# service kubelet restart
➜ root@cluster3-node1:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sun 2024-05-26 16:01:36 UTC; 7s ago
Docs: https://kubernetes.io/docs/
Main PID: 34514 (kubelet)
Tasks: 9 (limit: 1066)
Memory: 22.9M
CGroup: /system.slice/kubelet.service
└─34514 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/k>
...
Looking good, what does the node status say?
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster3-controlplane1 Ready control-plane 3h10m v1.30.1
cluster3-node1 Ready,SchedulingDisabled <none> 3h4m v1.30.1
Beautiful, let's make it schedulable again:
➜ k uncordon cluster3-node1
node/cluster3-node1 uncordoned
➜ k get node
NAME STATUS ROLES AGE VERSION
cluster3-controlplane1 Ready control-plane 3h10m v1.30.1
cluster3-node1 Ready <none> 3h4m v1.30.1
We're up to date.