API Certificates
If a new user, whether administrator or developer, needs access to the cluster, it's necessary to generate a certificate and sign it with the cluster's CA, the same one used to sign all other certificates of the cluster components.
As we saw earlier, the CA certificate is just a key pair (private key) and a self-signed certificate by that private key, but used to sign all other certificates in the cluster.
Whoever has access to this CA private key can sign any certificate and make it valid for the cluster, so it needs to be protected.
The process would be the same as generating a certificate for any cluster component: create the private key and a CSR, and send it to someone with access to this CA private key who can sign that certificate and send it back.
As we saw earlier, the certificate expires over time and the process to generate a new certificate is the same. This is how to have a certificate rotation workflow.
During CSR creation to be signed, we pass CN in the organization, so we can grant different privileges to the certificate.
Where can we find the CA private key? Inside the masters. But to avoid SSH access into a master by other users, we can use the Kubernetes API itself to do this signing. https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/
How would the process work using the API?
First, the user must generate their private key and the CSR to be signed by someone who has access to this API, in this case the administrator.
openssl genrsa -out david.key 2048
openssl req -new -key david.key -subj "/CN=david" -out david.csr
cat david.csr | base64 -w0
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
With this CSR we can now create an object in Kubernetes that signs this certificate. The CSR needs to be encrypted in base64 to be used in the object.
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: david
spec:
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
expirationSeconds: 31536000
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
- digital signature
- key encipherment
EOF
certificatesigningrequest.certificates.k8s.io/david created
Now someone will need to approve this request, in this case an administrator.
# Here we can see the CSRs and notice that the one we created is pending
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
david 5s kubernetes.io/kube-apiserver-client kubernetes-admin 365d Pending
# Let's approve it or we could deny using deny
kubectl certificate approve david
certificatesigningrequest.certificates.k8s.io/david approved
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
david 21s kubernetes.io/kube-apiserver-client kubernetes-admin 365d Approved,Issued
# If we look at the certificate we'll see that in the status we have the issued certificate but it's encoded in base64
kubectl get csr david -o yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"certificates.k8s.io/v1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"david"},"spec":{"expirationSeconds":31536000,"request":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=","signerName":"kubernetes.io/kube-apiserver-client","usages":["client auth","digital signature","key encipherment"]}}
creationTimestamp: "2024-02-09T11:58:53Z"
name: david
resourceVersion: "75265"
uid: 8c664716-83a3-4247-82a7-3a8800b69206
spec:
expirationSeconds: 31536000
groups:
- kubeadm:cluster-admins
- system:authenticated
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
- digital signature
- key encipherment
username: kubernetes-admin
status:
## Here we have our certificate, but encoded in base64
certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lSQU5HMkRoR2tERmNURGdmSTR0YTJvaG93RFFZSktvWklodmNOQVFFTEJRQXcKRlRFVE1CRUdBMVVFQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBeU1Ea3hNVFUwTVRoYUZ3MHlOVEF5TURneApNVFUwTVRoYU1CQXhEakFNQmdOVkJBTVRCV1JoZG1sa01JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBCk1JSUJDZ0tDQVFFQXlUMzJlUWFrcnB4V0ZzYUM4dTVTZUhoSkFVVktUNkgwdTNTTVJvY04va1p5M1UvcW9JRTkKYW5zUnN3Y3hDT28weE1DS3FwL0pzRzFDRnZvb2FWVVMyR0lSclp1bDhFdnRYcmMzc2d1NDR3UjZOM05VRTMyOAppMlZiWHFLUnhWbnAvSmkwRURRLytTQjB6TkJyZ0x0ZDRlbzN5cEpWOUhRajQ1VnI4STVkK0gyUE40TmRVRXhrCnFZUUVWU0diQWlYUndVeGRueUVQN2xIS05JcXVqZ2tJMkZqVHVSRWM1YmpWcHl5RmM4eFpORnN0MkZHZGk0SEwKZlNaTDNiWEhaNXh1akk3eWkxNUlKYlorbHpKb25XUld4SXd5NVVJSHg4WnpVd1BWcXR1SFFSYlZaWmIxcU52QQp5VzJDKzJWZStkMUhzNXpsVXF0YjEzT2s3cTJUVjZtaHpRSURBUUFCbzFZd1ZEQU9CZ05WSFE4QkFmOEVCQU1DCkJhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVcKZ0JRYUVRaFB3Qnoyd1dPbTY2RkVJN0hpRStWMUpqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFtM2NBNXRLZgpjYkd2VHBDTERTSnpPeElpbWJNOU4xOFRJQ0s2U2E1VXJuRzBKZ2JUV3RDdWRGZEhWOXFWakpLQ203WFZZM2djCnhDQXdzNUZ2L2FISVZTOTd1SGluc2d1dlRFMnlMU2xIUjlKMEtpY2l4MkoxaXBOSUhvK2wvaGQyY21iTGdhdFYKaFB5Sk1ab0JkVTk4ZUthd2pnb1VGczMvZStZU3IrclRmYU95WXVxSTJ0b1hlNUdLQ0VCdk02V1A5UDhOb00vLwpMUnZZZUxrZWhoOXBoUy9RTmZGY3VLeUNIakFodzBVTkNwWk5aalBjNm9jcVhkcVVXd000bHR6RW9OajFHU2hJCnZqUlpwSkVSY3NaQzFUR2xvYkYwUnlhWnAzZlEvNW1HaTI0c3ZIOFBmUmtvNWFvdVZ0WDRCU0FwdnRlWDVWdnAKYlJyY3J6VUlPVFZIQUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
conditions:
- lastTransitionTime: "2024-02-09T11:59:18Z"
lastUpdateTime: "2024-02-09T11:59:18Z"
message: This CSR was approved by kubectl certificate approve.
reason: KubectlApprove
status: "True"
type: Approved
# Let's get this value and create the certificate already decoded
kubectl get csr david -o jsonpath='{.status.certificate}' | base64 --decode > david.crt
# If we did echo "LS0tL........." | base64 --decode > david.crt it would be the same
# Here we'll verify the certificate
openssl x509 -in david.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
d1:b6:0e:11:a4:0c:57:13:0e:07:c8:e2:d6:b6:a2:1a
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes #<<<<
Validity
Not Before: Feb 9 11:54:18 2024 GMT
Not After : Feb 8 11:54:18 2025 GMT #<<<< 1 year validity
Subject: CN = david #<<<<
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c9:3d:f6:79:06:a4:ae:9c:56:16:c6:82:f2:ee:
52:78:78:49:01:45:4a:4f:a1:f4:bb:74:8c:46:87:
0d:fe:46:72:dd:4f:ea:a0:81:3d:6a:7b:11:b3:07:
31:08:ea:34:c4:c0:8a:aa:9f:c9:b0:6d:42:16:fa:
28:69:55:12:d8:62:11:ad:9b:a5:f0:4b:ed:5e:b7:
37:b2:0b:b8:e3:04:7a:37:73:54:13:7d:bc:8b:65:
5b:5e:a2:91:c5:59:e9:fc:98:b4:10:34:3f:f9:20:
74:cc:d0:6b:80:bb:5d:e1:ea:37:ca:92:55:f4:74:
23:e3:95:6b:f0:8e:5d:f8:7d:8f:37:83:5d:50:4c:
64:a9:84:04:55:21:9b:02:25:d1:c1:4c:5d:9f:21:
0f:ee:51:ca:34:8a:ae:8e:09:08:d8:58:d3:b9:11:
1c:e5:b8:d5:a7:2c:85:73:cc:59:34:5b:2d:d8:51:
9d:8b:81:cb:7d:26:4b:dd:b5:c7:67:9c:6e:8c:8e:
f2:8b:5e:48:25:b6:7e:97:32:68:9d:64:56:c4:8c:
32:e5:42:07:c7:c6:73:53:03:d5:aa:db:87:41:16:
d5:65:96:f5:a8:db:c0:c9:6d:82:fb:65:5e:f9:dd:
47:b3:9c:e5:52:ab:5b:d7:73:a4:ee:ad:93:57:a9:
a1:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
1A:11:08:4F:C0:1C:F6:C1:63:A6:EB:A1:44:23:B1:E2:13:E5:75:26
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
9b:77:00:e6:d2:9f:71:b1:af:4e:90:8b:0d:22:73:3b:12:22:
99:b3:3d:37:5f:13:20:22:ba:49:ae:54:ae:71:b4:26:06:d3:
5a:d0:ae:74:57:47:57:da:95:8c:92:82:9b:b5:d5:63:78:1c:
c4:20:30:b3:91:6f:fd:a1:c8:55:2f:7b:b8:78:a7:b2:0b:af:
4c:4d:b2:2d:29:47:47:d2:74:2a:27:22:c7:62:75:8a:93:48:
1e:8f:a5:fe:17:76:72:66:cb:81:ab:55:84:fc:89:31:9a:01:
75:4f:7c:78:a6:b0:8e:0a:14:16:cd:ff:7b:e6:12:af:ea:d3:
7d:a3:b2:62:ea:88:da:da:17:7b:91:8a:08:40:6f:33:a5:8f:
f4:ff:0d:a0:cf:ff:2d:1b:d8:78:b9:1e:86:1f:69:85:2f:d0:
35:f1:5c:b8:ac:82:1e:30:21:c3:45:0d:0a:96:4d:66:33:dc:
ea:87:2a:5d:da:94:5b:03:38:96:dc:c4:a0:d8:f5:19:28:48:
be:34:59:a4:91:11:72:c6:42:d5:31:a5:a1:b1:74:47:26:99:
a7:77:d0:ff:99:86:8b:6e:2c:bc:7f:0f:7d:19:28:e5:aa:2e:
56:d5:f8:05:20:29:be:d7:97:e5:5b:e9:6d:1a:dc:af:35:08:
39:35:47:00
And that's it!
Who does this signing? Although we used kube-apiserver to send the object, who actually does it is the controller-manager.
If we look more closely at what configuration we have in kube-controller-manager
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-name=kind-cluster
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key #<<<< It has the CA key
- --controllers=*,bootstrapsigner,tokencleaner
- --enable-hostpath-provisioner=true
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/16
- --use-service-account-credentials=true
...