Search code examples
kubernetescalico

Why does Kubernetes TCP networking require opened UDP ports?

I am running a self-managed Microk8s Kubernetes with Calico CNI. I have recently added a new node into the cluster and I have been seeing some network problems between pods on particular nodes.

I have been able to diagnose that the problem occurs when only TCP is allowed in the firewall rules. For some reason, when UDP traffic is not allowed through the firewall, intra-cluster TCP traffic stops working (I get timeouts from curl).

I am testing three different scenarios:

  1. Direct network traffic (one server to nginx on another server, no Kubernetes involved; Host -> Host)
  2. Request sent from one server to a NodePort service exposing a pod in on another server (Host -> Kubernetes)
  3. Request sent from a pod on one Node to a pod on another node (Kubernetes -> Kubernetes)

These are the results when only one of TCP or UDP protocols are allowed:

  • Firewall allows only TCP traffic
    • Host -> Host ✅
    • Host -> Kubernetes ✅
    • Kubernetes -> Kubernetes ❌
  • Firewall allows only UDP traffic
    • Host -> Host ❌
    • Host -> Kubernetes ❌
    • Kubernetes -> Kubernetes ✅

What is going on here? Somehow it seems that when intra-Kubernetes communication happens, the whole TCP communication is tunneled through UDP. I searched for Kubernetes or Calico needing the UDP ports to communicate but nothing eye-catching came up. Is this working as expected and do I need to always open both TCP and UDP?

Thank you

Solution

  • Calico uses VXLAN to create the overlay network, and VXLAN is a UDP protocol. From Wikipedia:

    Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to address the scalability problems associated with large cloud computing deployments. It uses a VLAN-like encapsulation technique to encapsulate OSI layer 2 Ethernet frames within layer 4 UDP datagrams, using 4789 as the default IANA-assigned destination UDP port number, although many implementations that predate the IANA assignment use port 8472. VXLAN endpoints, which terminate VXLAN tunnels and may be either virtual or physical switch ports, are known as VXLAN tunnel endpoints (VTEPs).