Hashes

We can verify the integrity of images used by containers through hashes.

I recommend reading hashes before continuing.

Let's compare the apiserver binary with the binary inside the container. We're on a cluster version 1.30.3. So we need to analyze everything in the document https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#downloads-for-v1303.

root@cks-master:~# wget https://dl.k8s.io/v1.30.3/kubernetes-server-linux-amd64.tar.gz
--2024-08-20 17:43:28--  https://dl.k8s.io/v1.30.3/kubernetes-server-linux-amd64.tar.gz
Resolving dl.k8s.io (dl.k8s.io)... 34.107.204.206, 2600:1901:0:26f3::
Connecting to dl.k8s.io (dl.k8s.io)|34.107.204.206|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cdn.dl.k8s.io/release/v1.30.3/kubernetes-server-linux-amd64.tar.gz [following]
--2024-08-20 17:43:28--  https://cdn.dl.k8s.io/release/v1.30.3/kubernetes-server-linux-amd64.tar.gz
Resolving cdn.dl.k8s.io (cdn.dl.k8s.io)... 151.101.193.55, 151.101.129.55, 151.101.65.55, ...
Connecting to cdn.dl.k8s.io (cdn.dl.k8s.io)|151.101.193.55|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 374416169 (357M) [application/x-tar]
Saving to: 'kubernetes-server-linux-amd64.tar.gz'

kubernetes-server-l 100%[===================>] 357.07M  89.4MB/s    in 4.0s

2024-08-20 17:43:32 (89.9 MB/s) - 'kubernetes-server-linux-amd64.tar.gz' saved [374416169/374416169]

# Saving the hash taken from the link above in compare

root@cks-master:~# echo "67282a349bd203fcc8d5d1d59d5b82fc56a14ea66f5a769ef457177ac5bcfb2fb65c239503a68f06a256f8919521fc96b5aad563bfec74eec13afb79a174f96b" > compare

# Also adding the hash we have to the compare file
root@cks-master:~# sha512sum kubernetes-server-linux-amd64.tar.gz | cut -d ' ' -f 1 >> compare
67282a349bd203fcc8d5d1d59d5b82fc56a14ea66f5a769ef457177ac5bcfb2fb65c239503a68f06a256f8919521fc96b5aad563bfec74eec13afb79a174f96b

# If we only have one line then it's the same.
root@cks-master:~# cat compare | uniq
67282a349bd203fcc8d5d1d59d5b82fc56a14ea66f5a769ef457177ac5bcfb2fb65c239503a68f06a256f8919521fc96b5aad563bfec74eec13afb79a174f96b

# Let's unzip to get the binary and go to it
root@cks-master:~# tar xzvf kubernetes-server-linux-amd64.tar.gz
root@cks-master:~# cd kubernetes/server/bin/

# Getting its sha512 and saving
root@cks-master:~/kubernetes/server/bin# ls kube-apiserver
kube-apiserver

root@cks-master:~/kubernetes/server/bin# ./kube-apiserver --version
Kubernetes v1.30.3

root@cks-master:~/kubernetes/server/bin# sha256sum kube-apiserver | cut -d ' ' -f 1 > apiserver-sha512

We now have access to the SHA256 hash of kube-apiserver now we need to check if it matches the hash of what we're running.

# If we try to enter the shell of the container in kube-apiserver we can't because it has no shell.
root@cks-master:/etc/kubernetes/manifests# k -n kube-system exec pods/kube-apiserver-cks-master -- sh
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "adc8bb808392966cfe63f7f425e72ec33c3494ea2e1d8d477741abcd19a875ba": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

# If we have the process number we can check the process filesystem and find the binary that's running, since we're on the node it's running on.

# The process number is 1479759
root@cks-master:~/kubernetes/server/bin# ps -aux | grep kube-apiserver
root     1479759  6.9  7.5 1541316 302980 ?      Ssl  18:10   0:50 kube-apiserver --advertise-address=10.128.0.5 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root     1485729  0.0  0.0   8172  2560 pts/0    S+   18:22   0:00 grep --color=auto kube-apiserver

# Looking at processes let's see this process's filesystem
root@cks-master:~/kubernetes/server/bin# ls /proc/1479759/root
bin  boot  dev  etc  go-runner  home  lib  proc  root  run  sbin  sys  tmp  usr  var

# Looking for kube-apiserver

root@cks-master:~/kubernetes/server/bin# find /proc/1479759/root/ | grep kube-apiserver
/proc/1479759/root/usr/local/bin/kube-apiserver

root@cks-master:~/kubernetes/server/bin# sha512sum /proc/1479759/root/usr/local/bin/kube-apiserver
e2a9bfddc7caa8139279b18b8c1588fc7f24fc87f44ec74a5878d90ab9c6d3003bdca81006058af0d189bd3ef2d99fc847e443a9242667129934f66960449ba3  /proc/1479759/root/usr/local/bin/kube-apiserver

# Taking only what we need and appending to the file
root@cks-master:~/kubernetes/server/bin# sha512sum /proc/1479759/root/usr/local/bin/kube-apiserver | cut -d ' ' -f 1
e2a9bfddc7caa8139279b18b8c1588fc7f24fc87f44ec74a5878d90ab9c6d3003bdca81006058af0d189bd3ef2d99fc847e443a9242667129934f66960449ba3

root@cks-master:~/kubernetes/server/bin# sha512sum /proc/1479759/root/usr/local/bin/kube-apiserver | cut -d ' ' -f 1 >> apiserver-sha512

# Confirming internal values and doing uniq
root@cks-master:~/kubernetes/server/bin# cat apiserver-sha512
e2a9bfddc7caa8139279b18b8c1588fc7f24fc87f44ec74a5878d90ab9c6d3003bdca81006058af0d189bd3ef2d99fc847e443a9242667129934f66960449ba3
e2a9bfddc7caa8139279b18b8c1588fc7f24fc87f44ec74a5878d90ab9c6d3003bdca81006058af0d189bd3ef2d99fc847e443a9242667129934f66960449ba3

root@cks-master:~/kubernetes/server/bin# cat apiserver-sha512 | uniq
e2a9bfddc7caa8139279b18b8c1588fc7f24fc87f44ec74a5878d90ab9c6d3003bdca81006058af0d189bd3ef2d99fc847e443a9242667129934f66960449ba3

Now let's move to a curiosity. When we do the command k get -n kube-system pod kube-apiserver-cks-master -o yaml we can observe the following block.

  containerStatuses:
  - containerID: containerd://84177449d2fcf44f6040a92914d5a5f9a2bb91c6bf341fb77026952b02092a30
    image: registry.k8s.io/kube-apiserver:v1.30.3
    # We have this sha256
    imageID: registry.k8s.io/kube-apiserver@sha256:a36d558835e48950f6d13b1edbe20605b8dfbc81e088f58221796631e107966c

This sha256 is from the IMAGE and not the binary. Let's check. Knowing that the files are the same whether we take the sha256 from the container binary or the one we downloaded. If the file is exactly the same and the function is the same then the hash is the same.

root@cks-master:~/kubernetes/server/bin# sha256sum kube-apiserver
286c16cf16389dfdfcd7e641859078c7b89f275c25e073d1c9f963cee393ccaa  kube-apiserver
root@cks-master:~/kubernetes/server/bin# sha256sum /proc/1479759/root/usr/local/bin/kube-apiserver
286c16cf16389dfdfcd7e641859078c7b89f275c25e073d1c9f963cee393ccaa  /proc/1479759/root/usr/local/bin/kube-apiserver

And we can see they are two different hashes.

# From the binary
286c16cf16389dfdfcd7e641859078c7b89f275c25e073d1c9f963cee393ccaa
 # From the Image
a36d558835e48950f6d13b1edbe20605b8dfbc81e088f58221796631e107966c

The kube-apiserver is normally built on an extremely minimalist base image, usually based on Linux distributions optimized for containers, such as distroless or scratch.

Common Options:

Distroless: is a set of base images created by Google that contain only what's necessary to run the application, without a shell, without package managers, and without any non-essential tools. Why use? These images are small, secure and focused on running the application with minimal overhead.
Scratch: Not exactly a base image, but an empty base, literally no operating system. Only static or extremely minimalist binaries are included. Why use? Scratch is the highest level of minimalism possible for containers, and kube-apiserver can be packaged this way if it's completely self-sufficient.

root@cks-master:~/kubernetes/server/bin# cat /proc/1479759/root/etc/os-release
PRETTY_NAME="Distroless"
NAME="Debian GNU/Linux"
ID="debian"
VERSION_ID="12"
VERSION="Debian GNU/Linux 12 (bookworm)"
HOME_URL="https://github.com/GoogleContainerTools/distroless"
SUPPORT_URL="https://github.com/GoogleContainerTools/distroless/blob/master/README.md"
BUG_REPORT_URL="https://github.com/GoogleContainerTools/distroless/issues/new"

So here's another security tip, reduce the attack surface of your microservices images using minimalist base images.