Question 18 - Fix Kubelet

Question 18 | Fix Kubelet

Use context: kubectl config use-context k8s-c3-CCC

There seems to be an issue with the kubelet not running on cluster3-node1. Fix it and confirm that cluster has node cluster3-node1 available in Ready state afterwards. You should be able to schedule a Pod on cluster3-node1 afterwards.

Write the reason of the issue into /opt/course/18/reason.txt.

Answer:

The procedure on tasks like these should be to check if the kubelet is running, if not start it, then check its logs and correct errors if there are some.

Always helpful to check if other clusters already have some of the components defined and running, so you can copy and use existing config files. Though in this case it might not need to be necessary.

kubectl config use-context k8s-c3-CCC
# Let's find which node has the problem
k get node
NAME                     STATUS     ROLES           AGE   VERSION
cluster3-controlplane1   Ready      control-plane   14d   v1.29.0
cluster3-node1           NotReady   <none>          14d   v1.29.0

## Let's enter the node and check the kubelet
ssh cluster3-node1
root@cluster3-node1:~# ps aux | grep kubelet
root     29294  0.0  0.2  14856  1016 pts/0    S+   11:30   0:00 grep --color=auto kubelet

# It's not running. We can check the service status
root@cluster3-node1:~# service kubelet status
# Or systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d

             └─10-kubeadm.conf
     Active: inactive (dead) (Result: exit-code) since Thu 2024-01-04 13:12:54 UTC; 1h 23min ago
       Docs: https://kubernetes.io/docs/
    Process: 27577 ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=>
   Main PID: 27577 (code=exited, status=203/EXEC)

Jan 04 13:12:52 cluster3-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jan 04 13:12:52 cluster3-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jan 04 13:12:54 cluster3-node1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

# It's inactive. Let's try to start it again
service kubelet start
# or systemctl start kuberlet
root@cluster3-node1:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2024-01-04 14:37:02 UTC; 6s ago
       Docs: https://kubernetes.io/docs/
    Process: 27935 ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=>
   Main PID: 27935 (code=exited, status=203/EXEC)

Jan 04 14:37:02 cluster3-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jan 04 14:37:02 cluster3-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

# Analyzing the problem above we have
    #Process: 27935 ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS #$KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=>
   #Main PID: 27935 (code=exited, status=203/EXEC)

# The call to kubelet is not coming up. If we were to execute kubelet we would need to do this
 root@cluster3-node1:~# /usr/local/bin/kubelet
-bash: /usr/local/bin/kubelet: No such file or directory
# This path doesn't point to kubelet

root@cluster3-node1:~# whereis kubelet
kubelet: /usr/bin/kubelet

# It's necessary to fix the binary path in
vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

# Now start again, but it's necessary to reload first
systemctl daemon-reload
service kubelet restart
service kubelet status  # Should be running now

# Going back to the default terminal

k get node
NAME                     STATUS   ROLES           AGE   VERSION
cluster3-controlplane1   Ready    control-plane   14d   v1.29.0
cluster3-node1           Ready    <none>          14d   v1.29.0

# The reason is that the path to kubelet is wrong. But let's write this in the file in English.
echo "wrong path to kubelet binary in service config" > /opt/course/18/reason.txt

Question 18 | Fix Kubelet​

Question 18 | Fix Kubelet