Skip to main content

Network Troubleshooting

CNI Plugins​

Various plugins can be used to create networking between nodes. In the exam, the URL will be provided if necessary. For more documentation https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy

  1. Weave Net: kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

  2. Flannel: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml 2.1. Does not support network policies.

  3. Calico: curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O; kubectl apply -f calico.yaml

If there are multiple CNI configuration files, kubelet will use the first one in alphabetical order.

DNS in Kubernetes​

As DNS server, Kubernetes uses CoreDNS which is very flexible and extensible.

Memory and Pods

The memory usage by CoreDNS is affected by the number of pods and services in the cluster, but only visible in large-scale clusters. Cache size and query rate also influence memory utilization.

The CoreDNS resources are:

Kubernetes resources for CoreDNS are:

  • service account: coredns
  • cluster-roles: coredns and kube-dns
  • clusterrolebindings: coredns and kube-dns
  • deployment: coredns
  • configmap: coredns
  • service: kube-dns

Analyzing the CoreDNS deployment, a configmap is used for configurations.

    spec:
      volumes:
        - name: config-volume
          configMap:
            name: coredns #<<<<
            items:
              - key: Corefile
                path: Corefile
            defaultMode: 420

Here's the configmap content.

kubectl describe cm -n kube-system coredns
Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
Corefile:
----
### Port 53 is used for DNS resolution.
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    # This is the k8s backend for cluster.local and reverse domains.
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    # Forward cluster domains directly to the correct authoritative DNS server.
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}


BinaryData
====

Events:  <none>

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

Troubleshooting issues related to CoreDNS.

  1. If CoreDNS is in pending state, check if any CNI plugin is installed.

  2. If CoreDNS pods are in CrashLoopBackOff or Error state. If you're using nodes running SELinux with an old Docker version, CoreDNS may not start. To solve this problem you can try the following steps.

    2.1. Update the container runtime version.

    2.2. Disable SELinux on the operating system.

    2.3. Modify the CoreDNS deployment and set allowPrivilegeEscalation to true.

    kubectl -n kube-system get deployment coredns -o yaml | sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | kubectl apply -f -

    2.4. Another cause for CoreDNS to have CrashLoopBackOff is when a CoreDNS Pod deployed in Kubernetes detects a loop. There are many ways to work around this issue, some are listed here:

    • Add the following to the kubelet yaml configuration: resolvConf: <path-to-your-real-resolv-conf-file> This flag tells kubelet to pass an alternative resolv.conf to pods. For systems that use systemd-resolved, /run/systemd/resolve/resolv.conf is typically the location of the "real" resolv.conf, although it may be different depending on your distribution.
    • Disable DNS local cache on host nodes and restore /etc/resolv.conf to the original.
    • A quick fix is to edit your Corefile, replacing forward . /etc/resolv.conf with the IP address of your upstream DNS, for example forward . 8.8.8.8. But this only fixes the CoreDNS problem, kubelet will continue forwarding the invalid resolv.conf to all default dnsPolicy pods, leaving them unable to resolve DNS.

  3. If CoreDNS pods and kube-dns service are working well, check if the kube-dns service has valid endpoints. kubectl -n kube-system get ep kube-dns If there are no endpoints for the service, inspect it and make sure it's using the correct selectors and ports.

Kube-Proxy​

kube-proxy is a network proxy that runs on each node in the cluster. kube-proxy maintains network rules on nodes. These network rules allow network communication to pods from network sessions inside or outside the cluster.

If the cluster was configured with kubeadm, kube-proxy will be running as a daemonset.

kube-proxy is responsible for monitoring services and endpoints associated with each service. When the client is going to connect to the service using the virtual IP, kube-proxy is responsible for sending traffic to the real pods.

If you run kubectl describe ds -n kube-system kube-proxy you can see that the kube-proxy binary is executed with the following command inside the kube-proxy container.

kubectl describe ds -n kube-system kube-proxy
Name:           kube-proxy
Selector:       k8s-app=kube-proxy
Node-Selector:  kubernetes.io/os=linux
Labels:         k8s-app=kube-proxy
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 4
Current Number of Nodes Scheduled: 4
Number of Nodes Scheduled with Up-to-date Pods: 4
Number of Nodes Scheduled with Available Pods: 4
Number of Nodes Misscheduled: 0
Pods Status:  4 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=kube-proxy
  Service Account:  kube-proxy
  Containers:
   kube-proxy:
    Image:      registry.k8s.io/kube-proxy:v1.29.1
    Port:       <none>
    Host Port:  <none>
####################
    Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      --hostname-override=$(NODE_NAME)
####################
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
  Volumes:
   kube-proxy:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-proxy
    Optional:  false
   xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
   lib-modules:
    Type:               HostPath (bare host directory volume)
    Path:               /lib/modules
    HostPathType:
  Priority Class Name:  system-node-critical
Events:                 <none>

So it fetches configuration from a /var/lib/kube-proxy/config.conf file and we can override the hostname with the node name on which the pod is running.

In the configuration file we define clusterCIDR, kubeproxy mode, ipvs, iptables, bindaddress, kube-config etc.

How to troubleshoot issues related to kube-proxy:

  1. Check if the kube-proxy pod in the kube-system namespace is running.

  2. Check kube-proxy logs.

  3. Check if the configmap is correctly defined and if the configuration file to run the kube-proxy binary is correct.

  4. kube-config is defined in the configmap.

  5. check if kube-proxy is running inside the container.

# netstat -plan | grep kube-proxy
tcp        0      0 0.0.0.0:30081           0.0.0.0:*               LISTEN      1/kube-proxy
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1/kube-proxy
tcp        0      0 172.17.0.12:33706       172.17.0.12:6443        ESTABLISHED 1/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      1/kube-proxy