Question 44 | CKS Challenge 4 - Security Monitoring & Incident Response

Take me to the lab!

Please note that the competition status for CKS Challenges is ended. Please do not submit a solution. It will not be scored.

There are a number of Kubernetes objects created inside the omega, citadel and eden-prime namespaces. However, several suspicious/abnormal operations have been observed in these namespaces!.

For example, in the citadel namespace, the application called webapp-color is constantly changing! You can see this for yourself by clicking on the citadel-webapp link and refreshing the page every 30 seconds. Similarly there are other issues with several other objects in other namespaces.

To understand what's causing these anomalies, you would be required to configure auditing in Kubernetes and make use of the Falco tool.

Inspect the issues in detail by clicking on the icons of the interactive architecture diagram in the lab and complete the tasks to secure the cluster. Once done click on the Check button to validate your work.

Do the tasks in this order

Click on all the icons individually and read the tasks. There is some important information within.

The Deployment icon in citadel namespace yields the following information:

Delete the rolebinding causing the constant deletion and creation of the configmaps and pods in this namespace.

So, this identifies the objects we need to audit in the next task.

Auditing/audit-log

The audit policy file should be stored at /etc/kubernetes/audit-policy.yaml
Create a single rule in the audit policy that will record events for the two objects depicting abnormal behaviour in the citadel namespace. This rule should however be applied to all three namespaces shown in the diagram at a metadata level. Omit the RequestReceived stage.
Use a volume called audit that will mount only the file /etc/kubernetes/audit-policy.yaml from the controlplane inside the api server pod in a read only mode.
audit-log-path set to /var/log/kubernetes/audit/audit.log

Create the audit policy.

vi /etc/kubernetes/audit-policy.yaml

Create the requested policy

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:             # Omit RequestReceived
    - RequestReceived
rules:
    - level: Metadata     # New rule at Metadata level
    resources:          # for pods and configmaps
    - group: ""
        resources:
        - pods
        - configmaps
    namespaces:         # in all three namespaces
    - omega
    - citadel
    - eden-prime

Mount the policy in api-server

Create the directory for the audit log first, or api-server will fail to come up

mkdir -p /var/log/kubernetes/audit

Edit api-server manifest

vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the required arguments to enable auditing

    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-policy-file=/etc/kubernetes/audit-policy.yaml

Add volumes (to any existing volumes) for the audit policy and log

    volumes:
    - name: audit-log
    hostPath:
        path: /var/log/kubernetes/audit/
        type: DirectoryOrCreate
    - name: audit
    hostPath:
        path: /etc/kubernetes/audit-policy.yaml
        type: File  # <- satifies requirement "will mount only the file"

Add volumeMounts (to any existing ones) for these volumes

    volumeMounts:
    - name: audit-log
        mountPath: var/log/kubernetes/audit/
        readOnly: false
    - name: audit
        mountPath: /etc/kubernetes/audit-policy.yaml
        readOnly: true      # <- The file should be immutable

Save and exit vi. Wait for api-server to restart. If it does not, know how to diagnose crashing API server.

Falco

Install the 'falco' utility on the controlplane node and start it as a systemd service

# Update apt indexes
apt-get update -y

# Install prerequiste and falco
apt-get -y install linux-headers-$(uname -r) falco

# Not strictly necessary to start it now, but if you want a green icon
# at this stage, you will need to start it.
systemctl start falco

Configure falco to save the event output to the file /opt/falco.log

open /etc/falco/falco.yaml in vi, find the file output section and make it like this

file_output:
    enabled: true
    keep_alive: false
    filename: /opt/falco.log

Reload falco

systemctl restart falco

security report

Inspect the API server audit logs and identify the user responsible for the abnormal behaviour seen in the citadel namespace. Save the name of the user, role and rolebinding responsible for the event to the file /opt/blacklist_users file (comma separated and in this specific order).
Inspect the falco logs and identify the pod that has events generated because of packages being updated on it. Save the namespace and the pod name in the file /opt/compromised_pods (comma separated - namespace followed by the pod name)

Inspect audit logs

Audit logs are JSON, one JSON record per line of the log file, and we know we are looking for citadel. Do a cursory scan of a few log lines to understand the structure. Use jq tool for format the log lines in a legible manner.

cat /var/log/kubernetes/audit/audit.log | grep citadel | head -4 | jq .

All the required information is likely there in the JSON you can see now, however let's improve the search with a jq filter to select delete events, since that is what we are looking for

cat /var/log/kubernetes/audit/audit.log | grep citadel | jq 'select (.verb == "delete")'

And there we have it. Pretty much all the records identify the perpetrator and the role/rolebinding being used.

Save results

echo 'agent-smith,important_role_do_not_delete,important_binding_do_not_delete' > /opt/blacklist_users

Inspect falco logs

Inspect logs

We've been told to look for something to do with packages:

grep -i package /opt/falco.log

Output:

19:23:46.797259642: Error Package management process launched in container (user=root user_loginuid=-1 command=apt install nginx container_id=55e02f53cced container_name=k8s_eden-software2_eden-software2_eden-prime_78092ae9-37b6-4a37-b01f-8b63c9598aa2_0 image=ubuntu:latest)

Identify pod

From the output (container_name=), we can determine

Namespace is eden-prime
Pod name is eden-software2

Save results

echo 'eden-prime,eden-software2' > /opt/compromised_pods

eden-prime/pod

Delete pods belonging to the eden-prime namespace that were flagged in the 'Security Report' file /opt/compromised_pods. Do not delete the non-compromised pods!

Using the pod discovered in the previous task with falco log:

kubectl delete pod -n eden-prime eden-software2

citadel/deploy

Delete the rolebinding causing the constant deletion and creation of the configmaps and pods in this namespace. Do not delete any other rolebinding!

Refer to what was found in the audit log

kubectl delete rolebinding -n citadel important_binding_do_not_delete

citadel/secret

Delete the role causing the constant deletion and creation of the configmaps and pods in this namespace. Do not delete any other role!

Refer to what was found in the audit log

kubectl delete role -n citadel important_role_do_not_delete

Auditing/audit-log​

Falco​

security report​

Inspect audit logs​

Inspect falco logs​

eden-prime/pod​

citadel/deploy​

citadel/secret​