API Certificates
Se um novo usuário seja administrador ou desenvolvedor precisar ter acesso ao cluster é necessário gerar um certificado e assiná-lo com a CA do cluster, a mesma usada para assinar todos os outros certificados dos componentes do cluster.
Como vimos anteriormente o CA certificado é somente um par de chaves private key e um certificado auto assinado pela própria chave privada, mas usada para assinar todos os outros certificados do cluster.
Quem tiver acesso a esta chave privada do CA poderá assinar qualquer certificado e torná-lo valido para o cluster, logo é necessário ser protegido.
O processo seria o mesmo que gerar um certificado para qualquer componenente do cluster, criar a chave privada, e um CSR e enviar para que alguém com acesso a esta chave privada do CA para possa assinar esse certificado e enviá-lo de volta.
Como vimos anteriormente o certificado expira com o tempo e o processo para gerar um novo certificado é o mesmo. Assim é o processo ter um fluxo de rotação dos certificados.
Durante a criação do csr para ser assinado, passamos CN na organização assim podemos conceder diferentes privilégios para o certificado.
Onde podemos encontrar esta a chave privada do CA? Dentro dos masters. Mas para evitar um acesso ssh para dentro de um master por outros usuários, podemos usar a própria API do kubernetes para fazer essa assinatura. https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/
Como seria o processo usando a API?
Primeiramente o usuário deve gerar sua chave privada e o csr para ser assinado por alguém que tem acesso a esta API, no caso o administrador.
openssl genrsa -out david.key 2048
openssl req -new -key david.key -subj "/CN=david" -out david.csr
cat david.csr | base64 -w0
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Com esse csr podemos criar agora um objeto no kubernetes que assine esse certificado. O csr precisa estar encriptado em base64 para ser usado no objeto.
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: david
spec:
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
expirationSeconds: 31536000
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
- digital signature
- key encipherment
EOF
certificatesigningrequest.certificates.k8s.io/david created
Agora alguém precisará aprovar esse request, no caso um administrador.
# Aqui podemos ver os csr e observe que o que criamos esta pendente
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
david 5s kubernetes.io/kube-apiserver-client kubernetes-admin 365d Pending
# Vamos approvar ou poderiamos negar usando o deny
kubectl certificate approve david
certificatesigningrequest.certificates.k8s.io/david approved
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
david 21s kubernetes.io/kube-apiserver-client kubernetes-admin 365d Approved,Issued
# Se observarmos o certificado veremos que no status temos o certificado emitido mas ele esta encodado em base 64.
kubectl get csr david -o yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"certificates.k8s.io/v1","kind":"CertificateSigningRequest","metadata":{"annotations":{},"name":"david"},"spec":{"expirationSeconds":31536000,"request":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=","signerName":"kubernetes.io/kube-apiserver-client","usages":["client auth","digital signature","key encipherment"]}}
creationTimestamp: "2024-02-09T11:58:53Z"
name: david
resourceVersion: "75265"
uid: 8c664716-83a3-4247-82a7-3a8800b69206
spec:
expirationSeconds: 31536000
groups:
- kubeadm:cluster-admins
- system:authenticated
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZUQ0NBVDBDQVFBd0VERU9NQXdHQTFVRUF3d0ZaR0YyYVdRd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQQpBNElCRHdBd2dnRUtBb0lCQVFESlBmWjVCcVN1bkZZV3hvTHk3bEo0ZUVrQlJVcFBvZlM3ZEl4R2h3MytSbkxkClQrcWdnVDFxZXhHekJ6RUk2alRFd0lxcW44bXdiVUlXK2locFZSTFlZaEd0bTZYd1MrMWV0emV5QzdqakJIbzMKYzFRVGZieUxaVnRlb3BIRldlbjhtTFFRTkQvNUlIVE0wR3VBdTEzaDZqZktrbFgwZENQamxXdndqbDM0Zlk4MwpnMTFRVEdTcGhBUlZJWnNDSmRIQlRGMmZJUS91VWNvMGlxNk9DUWpZV05PNUVSemx1TlduTElWenpGazBXeTNZClVaMkxnY3Q5Smt2ZHRjZG5uRzZNanZLTFhrZ2x0bjZYTW1pZFpGYkVqRExsUWdmSHhuTlRBOVdxMjRkQkZ0VmwKbHZXbzI4REpiWUw3WlY3NTNVZXpuT1ZTcTF2WGM2VHVyWk5YcWFITkFnTUJBQUdnQURBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFkTFF1emVGSVJDeUhGa3JXRUVjOEJiWHVUbFZ1Q1RLYVkvTm43VEczN2hkNmc2NWVxOHFICkxlMmZuNTJUbVdNb0hHTG9kZlB6WmYwY0ViNDF0WlprQnYxUGtQTDNzRTFjNmxmc1BXdFU5cUZZREN5RFcyQVAKUFJzNjJ0Z09VUDU0UitCVDhsWmtEQmpmSnlxS011L1p3MFdvUTQxcGV1SFlOdi9ESkN1VnpXenovdktxamY1agpDd1BSK2t3cllucnNKTHRKd3g5dzVlamNxaXl2bkVBVUFla0FjV2tZUGh4V3kyKzg1YlIzdFFKY3oweFo3TlFnCk90SHUrWEg2OWgrVFBlSXk5ZDhuTUNwMlV0R2prMzgrK1FnUVZ1eE92UnJ0SVl4c0lZVFU2VzNtTHFCQkpJckcKS3Q1UUdvUzlWMm81YlF6c3hhRkNTcjN4MGtWTWRQbDE4Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
signerName: kubernetes.io/kube-apiserver-client
usages:
- client auth
- digital signature
- key encipherment
username: kubernetes-admin
status:
## Aqui temos o nosso certificado, mas encodado em base64
certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lSQU5HMkRoR2tERmNURGdmSTR0YTJvaG93RFFZSktvWklodmNOQVFFTEJRQXcKRlRFVE1CRUdBMVVFQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBeU1Ea3hNVFUwTVRoYUZ3MHlOVEF5TURneApNVFUwTVRoYU1CQXhEakFNQmdOVkJBTVRCV1JoZG1sa01JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBCk1JSUJDZ0tDQVFFQXlUMzJlUWFrcnB4V0ZzYUM4dTVTZUhoSkFVVktUNkgwdTNTTVJvY04va1p5M1UvcW9JRTkKYW5zUnN3Y3hDT28weE1DS3FwL0pzRzFDRnZvb2FWVVMyR0lSclp1bDhFdnRYcmMzc2d1NDR3UjZOM05VRTMyOAppMlZiWHFLUnhWbnAvSmkwRURRLytTQjB6TkJyZ0x0ZDRlbzN5cEpWOUhRajQ1VnI4STVkK0gyUE40TmRVRXhrCnFZUUVWU0diQWlYUndVeGRueUVQN2xIS05JcXVqZ2tJMkZqVHVSRWM1YmpWcHl5RmM4eFpORnN0MkZHZGk0SEwKZlNaTDNiWEhaNXh1akk3eWkxNUlKYlorbHpKb25XUld4SXd5NVVJSHg4WnpVd1BWcXR1SFFSYlZaWmIxcU52QQp5VzJDKzJWZStkMUhzNXpsVXF0YjEzT2s3cTJUVjZtaHpRSURBUUFCbzFZd1ZEQU9CZ05WSFE4QkFmOEVCQU1DCkJhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3REFZRFZSMFRBUUgvQkFJd0FEQWZCZ05WSFNNRUdEQVcKZ0JRYUVRaFB3Qnoyd1dPbTY2RkVJN0hpRStWMUpqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFtM2NBNXRLZgpjYkd2VHBDTERTSnpPeElpbWJNOU4xOFRJQ0s2U2E1VXJuRzBKZ2JUV3RDdWRGZEhWOXFWakpLQ203WFZZM2djCnhDQXdzNUZ2L2FISVZTOTd1SGluc2d1dlRFMnlMU2xIUjlKMEtpY2l4MkoxaXBOSUhvK2wvaGQyY21iTGdhdFYKaFB5Sk1ab0JkVTk4ZUthd2pnb1VGczMvZStZU3IrclRmYU95WXVxSTJ0b1hlNUdLQ0VCdk02V1A5UDhOb00vLwpMUnZZZUxrZWhoOXBoUy9RTmZGY3VLeUNIakFodzBVTkNwWk5aalBjNm9jcVhkcVVXd000bHR6RW9OajFHU2hJCnZqUlpwSkVSY3NaQzFUR2xvYkYwUnlhWnAzZlEvNW1HaTI0c3ZIOFBmUmtvNWFvdVZ0WDRCU0FwdnRlWDVWdnAKYlJyY3J6VUlPVFZIQUE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
conditions:
- lastTransitionTime: "2024-02-09T11:59:18Z"
lastUpdateTime: "2024-02-09T11:59:18Z"
message: This CSR was approved by kubectl certificate approve.
reason: KubectlApprove
status: "True"
type: Approved
# Vamos pegar este valor e criar o certificado já desencodando
kubectl get csr david -o jsonpath='{.status.certificate}' | base64 --decode > david.crt
# Se fizessemos echo "LS0tL........." | base64 --decode > david.crt daria no mesmo
# Aqui vamos conferir o certificado
openssl x509 -in david.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
d1:b6:0e:11:a4:0c:57:13:0e:07:c8:e2:d6:b6:a2:1a
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = kubernetes #<<<<
Validity
Not Before: Feb 9 11:54:18 2024 GMT
Not After : Feb 8 11:54:18 2025 GMT #<<<< validade de 1 ano
Subject: CN = david #<<<<
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c9:3d:f6:79:06:a4:ae:9c:56:16:c6:82:f2:ee:
52:78:78:49:01:45:4a:4f:a1:f4:bb:74:8c:46:87:
0d:fe:46:72:dd:4f:ea:a0:81:3d:6a:7b:11:b3:07:
31:08:ea:34:c4:c0:8a:aa:9f:c9:b0:6d:42:16:fa:
28:69:55:12:d8:62:11:ad:9b:a5:f0:4b:ed:5e:b7:
37:b2:0b:b8:e3:04:7a:37:73:54:13:7d:bc:8b:65:
5b:5e:a2:91:c5:59:e9:fc:98:b4:10:34:3f:f9:20:
74:cc:d0:6b:80:bb:5d:e1:ea:37:ca:92:55:f4:74:
23:e3:95:6b:f0:8e:5d:f8:7d:8f:37:83:5d:50:4c:
64:a9:84:04:55:21:9b:02:25:d1:c1:4c:5d:9f:21:
0f:ee:51:ca:34:8a:ae:8e:09:08:d8:58:d3:b9:11:
1c:e5:b8:d5:a7:2c:85:73:cc:59:34:5b:2d:d8:51:
9d:8b:81:cb:7d:26:4b:dd:b5:c7:67:9c:6e:8c:8e:
f2:8b:5e:48:25:b6:7e:97:32:68:9d:64:56:c4:8c:
32:e5:42:07:c7:c6:73:53:03:d5:aa:db:87:41:16:
d5:65:96:f5:a8:db:c0:c9:6d:82:fb:65:5e:f9:dd:
47:b3:9c:e5:52:ab:5b:d7:73:a4:ee:ad:93:57:a9:
a1:cd
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
1A:11:08:4F:C0:1C:F6:C1:63:A6:EB:A1:44:23:B1:E2:13:E5:75:26
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
9b:77:00:e6:d2:9f:71:b1:af:4e:90:8b:0d:22:73:3b:12:22:
99:b3:3d:37:5f:13:20:22:ba:49:ae:54:ae:71:b4:26:06:d3:
5a:d0:ae:74:57:47:57:da:95:8c:92:82:9b:b5:d5:63:78:1c:
c4:20:30:b3:91:6f:fd:a1:c8:55:2f:7b:b8:78:a7:b2:0b:af:
4c:4d:b2:2d:29:47:47:d2:74:2a:27:22:c7:62:75:8a:93:48:
1e:8f:a5:fe:17:76:72:66:cb:81:ab:55:84:fc:89:31:9a:01:
75:4f:7c:78:a6:b0:8e:0a:14:16:cd:ff:7b:e6:12:af:ea:d3:
7d:a3:b2:62:ea:88:da:da:17:7b:91:8a:08:40:6f:33:a5:8f:
f4:ff:0d:a0:cf:ff:2d:1b:d8:78:b9:1e:86:1f:69:85:2f:d0:
35:f1:5c:b8:ac:82:1e:30:21:c3:45:0d:0a:96:4d:66:33:dc:
ea:87:2a:5d:da:94:5b:03:38:96:dc:c4:a0:d8:f5:19:28:48:
be:34:59:a4:91:11:72:c6:42:d5:31:a5:a1:b1:74:47:26:99:
a7:77:d0:ff:99:86:8b:6e:2c:bc:7f:0f:7d:19:28:e5:aa:2e:
56:d5:f8:05:20:29:be:d7:97:e5:5b:e9:6d:1a:dc:af:35:08:
39:35:47:00
E é isso!
Quem faz essa assinatura? Apesar de ter usado o kube-apiserver para enviar o objeto, quem de fato faz é o controller-manager.
Se olharmos mais de perto o que temos de configuração no kube-controller-manager
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --allocate-node-cidrs=true
- --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
- --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --cluster-cidr=10.244.0.0/16
- --cluster-name=kind-cluster
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key #<<<< Ele tem a chave do CA
- --controllers=*,bootstrapsigner,tokencleaner
- --enable-hostpath-provisioner=true
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/16
- --use-service-account-credentials=true
...