Skip to main content

Question 2 | Runtime Security with Falco

Use context: kubectl config use-context workload-prod

Falco is installed with default configuration on node cluster1-node1. Connect using ssh cluster1-node1. Use it to:

Find a Pod running image nginx which creates unwanted package management processes inside its container.

Find a Pod running image httpd which modifies /etc/passwd.

Save the Falco logs for case 1 under /opt/course/2/falco.log in format:

time-with-nanosconds,container-id,container-name,user-name No other information should be in any line. Collect the logs for at least 30 seconds.

Afterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.

Answer:

Falco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine.

NOTE: Other tools you might have to be familar with are sysdig or tracee

Use Falco as service

First we can investigate Falco config a little:

➜ ssh cluster1-node1

➜ root@cluster1-node1:~# service falco status
● falco.service - LSB: Falco syscall activity monitoring agent
   Loaded: loaded (/etc/init.d/falco; generated)
   Active: active (running) since Sat 2020-10-10 06:36:15 UTC; 2h 1min ago
...

➜ root@cluster1-node1:~# cd /etc/falco

➜ root@cluster1-node1:/etc/falco# ls
falco.yaml  falco_rules.local.yaml  falco_rules.yaml  k8s_audit_rules.yaml  rules.available  rules.d

This is the default configuration, if we look into falco.yaml we can see:

# /etc/falco/falco.yaml

...
# Where security notifications should go.
# Multiple outputs can be enabled.

syslog_output:
  enabled: true
...

This means that Falco is writing into syslog, hence we can do:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco
Sep 15 08:44:04 ubuntu2004 falco: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Sep 15 08:44:04 ubuntu2004 falco: Falco initialized with configuration file /etc/falco/falco.yaml
Sep 15 08:44:04 ubuntu2004 falco: Loading rules from file /etc/falco/falco_rules.yaml:
...

Yep, quite some action going on in there. Let's investigate the first offending Pod:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep nginx | grep process
Sep 16 06:23:47 ubuntu2004 falco: 06:23:47.376241377: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)
...

➜ root@cluster1-node1:~# crictl ps -id 7a5ea6a080d1
CONTAINER ID        IMAGE              NAME        ...         POD ID
7a5ea6a080d1b       6f715d38cfe0e      nginx       ...         7a864406b9794

root@cluster1-node1:~# crictl pods -id 7a864406b9794
POD ID              ...          NAME                             NAMESPACE        ...
7a864406b9794       ...          webapi-6cfddcd6f4-ftxg4          team-blue        ...

First Pod is webapi-6cfddcd6f4-ftxg4 in Namespace team-blue.

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep httpd | grep passwd
Sep 16 06:23:48 ubuntu2004 falco: 06:23:48.830962378: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sed -i $d /etc/passwd parent=sh pcmdline=sh -c echo hacker >> /etc/passwd; sed -i '$d' /etc/passwd; true file=/etc/passwdngFmAl program=sed gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=b1339d5cc2de image=docker.io/library/httpd)

➜ root@cluster1-node1:~# crictl ps -id b1339d5cc2de
CONTAINER ID        IMAGE              NAME        ...         POD ID
b1339d5cc2dee       f6b40f9f8ad71      httpd       ...         595af943c3245

root@cluster1-node1:~# crictl pods -id 595af943c3245
POD ID              ...          NAME                             NAMESPACE          ...
595af943c3245       ...          rating-service-68cbdf7b7-v2p6g   team-purple        ...

Second Pod is rating-service-68cbdf7b7-v2p6g in Namespace team-purple.

Eliminate offending Pods

The logs from before should allow us to find and "eliminate" the offending Pods:

➜ k get pod -A | grep webapi
team-blue              webapi-6cfddcd6f4-ftxg4                      1/1     Running

➜ k -n team-blue scale deploy webapi --replicas 0
deployment.apps/webapi scaled

➜ k get pod -A | grep rating-service
team-purple            rating-service-68cbdf7b7-v2p6g               1/1     Running

➜ k -n team-purple scale deploy rating-service --replicas 0
deployment.apps/rating-service scaled

Use Falco from command line

We can also use Falco directly from command line, but only if the service is disabled:

➜ root@cluster1-node1:~# service falco stop

➜ root@cluster1-node1:~# falco
Thu Sep 16 06:33:11 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Thu Sep 16 06:33:11 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Thu Sep 16 06:33:12 2021: Starting internal webserver, listening on port 8765
06:33:17.382603204: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)
...

We can see that rule files are loaded and logs printed afterwards.

Create logs in correct format

The task requires us to store logs for "unwanted package management processes" in format time,container-id,container-name,user-name. The output from falco shows entries for "Error Package management process launched" in a default format. Let's find the proper file that contains the rule and change it:

➜ root@cluster1-node1:~# cd /etc/falco/

➜ root@cluster1-node1:/etc/falco# grep -r "Package management process launched" .
./falco_rules.yaml:    Package management process launched in container (user=%user.name user_loginuid=%user.loginuid

➜ root@cluster1-node1:/etc/falco# cp falco_rules.yaml falco_rules.yaml_ori

➜ root@cluster1-node1:/etc/falco# vim falco_rules.yaml

Find the rule which looks like this:

# Container is supposed to be immutable. Package management should be done in building the image.
- rule: Launch Package Management Process in Container
  desc: Package management process ran inside container
  condition: >
    spawned_process
    and container
    and user.name != "_apt"
    and package_mgmt_procs
    and not package_mgmt_ancestor_procs
    and not user_known_package_manager_in_container
  output: >
    Package management process launched in container (user=%user.name user_loginuid=%user.loginuid
    command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)
  priority: ERROR
  tags: [process, mitre_persistence]

Should be changed into the required format:

# Container is supposed to be immutable. Package management should be done in building the image.
- rule: Launch Package Management Process in Container
  desc: Package management process ran inside container
  condition: >
    spawned_process
    and container
    and user.name != "_apt"
    and package_mgmt_procs
    and not package_mgmt_ancestor_procs
    and not user_known_package_manager_in_container
  output: >
    Package management process launched in container %evt.time,%container.id,%container.name,%user.name
  priority: ERROR
  tags: [process, mitre_persistence]
For all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.

Next we check the logs in our adjusted format:

➜ root@cluster1-node1:/etc/falco# falco | grep "Package management"

06:38:28.077150666: Error Package management process launched in container 06:38:28.077150666,090aad374a0a,nginx,root
06:38:33.058263010: Error Package management process launched in container 06:38:33.058263010,090aad374a0a,nginx,root
06:38:38.068693625: Error Package management process launched in container 06:38:38.068693625,090aad374a0a,nginx,root
06:38:43.066159360: Error Package management process launched in container 06:38:43.066159360,090aad374a0a,nginx,root
06:38:48.059792139: Error Package management process launched in container 06:38:48.059792139,090aad374a0a,nginx,root
06:38:53.063328933: Error Package management process launched in container 06:38:53.063328933,090aad374a0a,nginx,root

This looks much better. Copy&paste the output into file /opt/course/2/falco.log on your main terminal. The content should be cleaned like this:

# /opt/course/2/falco.log
06:38:28.077150666,090aad374a0a,nginx,root
06:38:33.058263010,090aad374a0a,nginx,root
06:38:38.068693625,090aad374a0a,nginx,root
06:38:43.066159360,090aad374a0a,nginx,root
06:38:48.059792139,090aad374a0a,nginx,root
06:38:53.063328933,090aad374a0a,nginx,root
06:38:58.070912841,090aad374a0a,nginx,root
06:39:03.069592140,090aad374a0a,nginx,root
06:39:08.064805371,090aad374a0a,nginx,root
06:39:13.078109098,090aad374a0a,nginx,root
06:39:18.065077287,090aad374a0a,nginx,root
06:39:23.061012151,090aad374a0a,nginx,root

For a few entries it should be fast to just clean it up manually. If there are larger amounts of entries we could do:

cat /opt/course/2/falco.log.dirty | cut -d" " -f 9 > /opt/course/2/falco.log

The tool cut will split input into fields using space as the delimiter (-d""). We then only select the 9th field using -f 9.

Local falco rules

There is also a file /etc/falco/falco_rules.local.yaml in which we can override existing default rules. This is a much cleaner solution for production. Choose the faster way for you in the exam if nothing is specified in the task.