Cloud Platform Node Metadata

A metadata server in cloud platforms is a service that provides specific information about a virtual machine (VM) instance or runtime environment. It is accessed from VMs to obtain data useful for configuring and operating the instance itself.

Instance Information:
- Instance ID: Unique identification of the VM in the cloud infrastructure.
- Instance Type: Machine specifications, such as the VM family and size.
- Availability Zone: Information about the geographic location or zone where the VM is running.
- Hostname: Machine or instance host name.
Configuration Data:
- Initialization Scripts: Scripts or commands that should be executed when the instance starts.
- SSH Keys: Public keys used for SSH access to the instance.
- Tags and Labels: Tags and labels that can be used for VM identification or classification.
Temporary Credentials:
- Temporary Access Keys: Temporary access credentials for the cloud API, allowing the instance to authenticate and access other resources.
- Identity Tokens: Tokens for authentication and authorization in cloud services.
Network Data:
- IP Addresses: Public and private IP of the instance.
- Network Gateway: Network gateway address for network egress.
- DNS: DNS servers configured for the instance.
Dynamic Customization:
- Custom Data: Custom data that can be passed to the instance at creation time and used by software or scripts for customization.
- User Data: Similar to custom data, can contain scripts or commands that are executed during instance boot.
Security Information:
- Instance Identity: Information about the instance identity for services like AWS IAM, Google Cloud IAM, or Azure AD.
- SSL/TLS Certificates: Certificates and keys used for secure communication.
Examples of Metadata Servers in Cloud Platforms:
- AWS (Amazon Web Services): http://169.254.169.254/latest/meta-data/
- Google Cloud: http://169.254.169.254/computeMetadata/v1/
- Azure: http://169.254.169.254/metadata/instance?api-version=2021-02-01
Usefulness:
- Automation: Allows automating the configuration and management of VMs dynamically.
- Security: Provides temporary credentials and identity information securely.
- Flexibility: Allows instances to dynamically adjust to their configurations and operational environment.

These metadata servers are fundamental for efficient and secure operation of VMs in the cloud, providing a centralized and standardized way to access critical instance data.

When we set up a Kubernetes cluster from VMs not managed by the cloud, we need to worry about this security and prevent pods running inside a VM from accessing these servers.

To mitigate this we must:

Ensure that the account used by instances only has the necessary permissions, but this is outside the scope of Kubernetes.
Follow the set of recommendations from each cloud provider, which is also outside the scope of Kubernetes, but it's good to be aware.

What we can do is block using Network Policies. We can block everything and only allow what we want.

In our environment we have a master and a worker, and since we're using gcloud, let's try to access the gcloud server metadata directly from the node and from inside a container of a pod we're running.

# See that we can get the image it's using directly from the node
root@cks-master:~# curl "http://metadata.google.internal/computeMetadata/v1/instance/image" -H "Metadata-Flavor: Google"
projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240808root@cks-master:~#

root@cks-master:~# k get pods
NAME   READY   STATUS    RESTARTS   AGE
app1   1/1     Running   0          6h58m
app2   1/1     Running   0          6h58m

# Running the same command in the app1 pod container we also can.
root@cks-master:~# k exec app1 -- curl "http://metadata.google.internal/computeMetadata/v1/instance/image" -H "Metadata-Flavor: Google"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    66  100    66    0     0   1946      0 --:--:--projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240808 --:--:-- --:--:--  2000

We can block this by knowing the IP and creating a network policy to deny everything.

root@cks-master:~# ping metadata.google.internal
PING metadata.google.internal (169.254.169.254) 56(84) bytes of data.
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=1 ttl=255 time=3.88 ms
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=2 ttl=255 time=3.72 ms
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=3 ttl=255 time=4.11 ms
64 bytes from metadata.google.internal (169.254.169.254): icmp_seq=4 ttl=255 time=4.21 ms
^C
--- metadata.google.internal ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 3.718/3.978/4.208/0.192 ms
# We already have the IP

Now just create a rule allowing internet egress but blocking the specific IP. This rule will be applied to all pods in the namespace.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cloud-metadata-deny
  namespace: default
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0 # Allowing internet egress
        except: # Except this IP
        - 169.254.169.254/32

root@cks-master:~/default# kubectl apply -f np-deny-node-metadata.yaml
networkpolicy.networking.k8s.io/cloud-metadata-deny created

# And the pod no longer has access
root@cks-master:~/default# k exec app1 -- curl "http://metadata.google.internal/computeMetadata/v1/instance/image" -H "Metadata-Flavor: Google"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0^C

# But the node still has.
root@cks-master:~/default# curl "http://metadata.google.internal/computeMetadata/v1/instance/image" -H "Metadata-Flavor: Google"
projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240808root@cks-master:~/default#

If we need a pod that has access to the cloud metadata server, we create a specific rule for it, stating what it can access.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cloud-metadata-allow
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: metadata-accessor # Allow the pod with this label
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 169.254.169.254/32 # We are allowing this IP

Note that we are applying two rules to the same pod. One for all pods including the one we want to allow and another allowing only this one. With the union between these rules, the rule that allows has preference.