Secrets
Although it's possible to store information using ConfigMaps, it's not the ideal approach for sensitive data, such as passwords.
For these cases, it's recommended to use Secrets, specifically designed to store critical information, such as passwords or keys, just like ConfigMap, but with additional security layers.
Secrets work exactly like ConfigMaps. First we must create them to be used in a way that they are injected into other objects.
We have two methods for creating Secrets.
- Imperative: When we use the command line directly.
- Declarative: When we use files.
A Secret object is exactly the same as a ConfigMap object in its form, but the data needs to be base64 encoded.
apiVersion: v1
kind: Secret
metadata:
data:
To create a secret using imperative mode we can do:
# Generic is the type of secret
kubectl create secret generic mysecret --from-literal=key1=david,key2=david2
# Note that the value is not shown
kubectl describe secrets mysecret
Name: mysecret
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
key1: 17 bytes
# To delete
kubectl delete secrets mysecret
secret "mysecret" deleted
During the command the key values are automatically encoded.
If we want to create this with files we need to pass the keys with base64 encoded value.
echo -n "david" | base64
ZGF2aWQ=
# just to show how to decode
echo "ZGF2aWQ=" | base64 --decode
david
echo -n "david2" | base64
ZGF2aWQy
echo "ZGF2aWQy" | base64 --decode
david2
apiVersion: v1
kind: Secret
metadata:
name: mysecret
data:
key1: ZGF2aWQ=
key2: ZGF2aWQy
kubectl apply -f mysecret.yaml
secret/mysecret created
kubectl get secrets
NAME TYPE DATA AGE
mysecret Opaque 2 22s
sh.helm.release.v1.istio-base.v1 helm.sh/release.v1 1 12d
sh.helm.release.v1.metrics-server.v1 helm.sh/release.v1 1 12h
If we want to see the opaque values we need to get the complete yaml output.
kubectl get secrets mysecret -o yaml
apiVersion: v1
data:
key1: ZGF2aWQ= #<<<<
key2: ZGF2aWQy #<<<<
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"key1":"ZGF2aWQ=","key2":"ZGF2aWQy"},"kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"}}
creationTimestamp: "2024-03-18T14:16:06Z"
name: mysecret
namespace: default
resourceVersion: "2824576"
uid: 20429ec4-c03c-4184-ae0f-
With these values we need to decode as shown previously.
Now let's use this secret in a pod. It would be exactly as we would do with configmap, see the difference.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
run: echo
name: echo
spec:
containers:
- image: alpine
name: echo
command: ["sleep"]
args:
- "1000"
# envFrom:
# - configMapRef:
# name: myconfigmap
envFrom:
- secretRef:
name: mysecret
dnsPolicy: ClusterFirst
restartPolicy: Always
EOF
kubectl exec echo -- env | grep key
key1=david
key2=david2
In this case we are injecting the value of all keys key1=david and key2=david2. To add only one of these keys we can do.
To get only one key from the secret that contains several we can do.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
run: echo2
name: echo2
spec:
containers:
- image: alpine
name: echo
command: ["sleep"]
args:
- "1000"
env:
- name: mykey
valueFrom:
secretKeyRef:
name: mysecret
key: key2
dnsPolicy: ClusterFirst
restartPolicy: Always
EOF
kubectl exec echo2 -- env | grep key
mykey=david2
We can also inject the secret as files.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
run: echo3
name: echo3
spec:
containers:
- image: alpine
name: echo
command: ["sleep"]
args:
- "1000"
volumeMounts:
- name: mysecret-volume
mountPath: /mysecrets
volumes:
- name: mysecret-volume
secret:
secretName: mysecret
EOF
kubectl exec echo3 -- ls /mysecrets
key1
key2
kubectl exec echo3 -- cat /mysecrets/key1
david%
kubectl exec echo3 -- cat /mysecrets/key2
david2%
In the example above we are creating a volume with the key values and these will be used in the volume mount. We'll have /mysecrets/key1 and /mysecrets/key2 and the content will be the real values of the keys, i.e., not encoded.
Since this pod is on worker3, let's check what happens inside.
k get pods echo3 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
echo3 1/1 Running 0 5m26s 10.44.0.6 cka-cluster-worker3 <none> <none>
kubectl get pods echo3 -o yaml | grep uid
uid: 08198260-0b94-4934-93a3-b63d252163a7
# THE COMMANDS BELOW WILL BE INSIDE WORKER3 WHERE THE POD WENT
root@cka-cluster-worker3:/opt# cd /var/lib/kubelet/pods/
root@cka-cluster-worker3:/var/lib/kubelet/pods# ls
08198260-0b94-4934-93a3-b63d252163a7 576a9764-b33e-48e4-82d3-daa925c59c0b b055b81d-96d4-4fe7-8666-635a9d03f4bf e9870bb1-96dd-4ca2-b2e0-885cac2d26f8 fe715166-b49a-4e50-8e5b-1b6e3b65d5b0
50b04dc3-709a-40fa-a916-4f78d3f3ce1b 67a3c831-9b96-4987-8b98-4695e90b0322 c9626cf9-9095-48e0-b313-e1b8f616a0b8 eb4b3290-e36a-4a8e-9d28-39b0cfbe4ac1
# Same uid
root@cka-cluster-worker3:/var/lib/kubelet/pods# cd 08198260-0b94-4934-93a3-b63d252163a7/
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7# ls
containers etc-hosts plugins volumes
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7# cd volumes/
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes# ls
kubernetes.io~projected kubernetes.io~secret
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes# cd kubernetes.io~secret/
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes/kubernetes.io~secret# ls
mysecret-volume
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes/kubernetes.io~secret# cd mysecret-volume/
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes/kubernetes.io~secret/mysecret-volume# ls
key1 key2
root@cka-cluster-worker3:/var/lib/kubelet/pods/08198260-0b94-4934-93a3-b63d252163a7/volumes/kubernetes.io~secret/mysecret-volume# cat key1
david
Some details about secrets:
- Secrets are not encrypted, but encoded. Anyone can simply decode a secret in base64 during the code just as we did manually.
- Secrets are not stored encrypted in etcd, but it's possible to do encryption at rest.
- Anyone with permission to create pods/deployments in the same namespace as the secrets can access the secrets. It's necessary to ensure privileges to access secrets using RBAC.
- It's good to consider other alternatives for storing like AWS, Azure, GCP, Vault, etc
Encryption at Rest
It's possible to encrypt data when it's written to ETCD. This doesn't mean that data after being read from ETCD is encrypted.
For this to be possible, kube-apiserver needs to know what should be encrypted or not.
In kube-apiserver we have the passed configurations.
kubectl get pods -n kube-system kube-apiserver-cka-cluster-control-plane -o=jsonpath='{.spec.containers[*].command}' | jq
[
"kube-apiserver",
"--advertise-address=172.18.0.5",
"--allow-privileged=true",
"--authorization-mode=Node,RBAC",
"--client-ca-file=/etc/kubernetes/pki/ca.crt",
"--enable-admission-plugins=NodeRestriction",
"--enable-bootstrap-token-auth=true",
"--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt",
"--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt",
"--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key",
"--etcd-servers=https://127.0.0.1:2379",
"--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt",
"--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key",
"--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
"--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt",
"--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key",
"--requestheader-allowed-names=front-proxy-client",
"--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt",
"--requestheader-extra-headers-prefix=X-Remote-Extra-",
"--requestheader-group-headers=X-Remote-Group",
"--requestheader-username-headers=X-Remote-User",
"--runtime-config=",
"--secure-port=6443",
"--service-account-issuer=https://kubernetes.default.svc.cluster.local",
"--service-account-key-file=/etc/kubernetes/pki/sa.pub",
"--service-account-signing-key-file=/etc/kubernetes/pki/sa.key",
"--service-cluster-ip-range=10.96.0.0/12",
"--tls-cert-file=/etc/kubernetes/pki/apiserver.crt",
"--tls-private-key-file=/etc/kubernetes/pki/apiserver.key"
]
--encription-provider-config was not passed.
Let's first create this configuration file to know what should or shouldn't be encrypted.
It's not necessary to encrypt all resources, only those that have sensitive data, for example secrets and configmaps.
# We need to generate an encryption key with the command below. This key will be used to encrypt data going to etcd.
ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
echo $ENCRYPTION_KEY
5T5/bTFcDZYzYisA4DC1mw1Qpl7NIP4BOhMkfM7zgz8=
cat > encryption-config.yaml <<EOF
apiVersion: v1
kind: EncryptionConfig
resources:
- resources: # What resources do we want to encrypt??
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret: ${ENCRYPTION_KEY}
- identity: {}
EOF
The providers are those used to encrypt resources. The order they are passed matters. If identity which has nothing is first, nothing will be encrypted.
Put the file inside /etc/kubernetes for example which is already mounted and reference the path to this file.
From here on all configmaps and secrets resources are encrypted in ETCD.
If you already have secrets and need to encrypt them, we need to recreate them.
kubectl get secrets --all-namespaces -o yaml > allsecrets.yaml
kubectl replace --force -f allsecrets.yaml