Search code examples
ubuntukubernetesiptablesk3s

k3s networking problems on Ubuntu 22.04: failed to get Kubernetes server version: Get \"https://10.43.0.1:443/version?timeout=32s\": i/o timeout

After upgrading from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS, I am currently facing issues with my k3s cluster.

For instance, the logs of the local-path-provisioner pod:

$ kubectl logs -n kube-system local-path-provisioner-6c79684f77-l4cqp
time="2022-04-28T03:27:00Z" level=fatal msg="Error starting daemon: Cannot start Provisioner: failed to get Kubernetes server version: Get \"https://10.43.0.1:443/version?timeout=32s\": dial tcp 10.43.0.1:443: i/o timeout"

I've tried the following actions:

  • Disabling ipv6, as described here
  • Disabling ufw firewall
  • Use legacy iptables
  • Adding rules for internal traffic to iptables, like so:
$ sudo iptables -A INPUT -s 10.42.0.0/16 -d <host_ip> -j ACCEPT

Still, coredns, local-path-provisioner and metrics-server deployments won't start. When listing pods, here's the output:

$ kubectl get pods -n kube-system                                    
NAME                                      READY   STATUS             RESTARTS         AGE
cilium-64r4c                              1/1     Running            2 (18m ago)      174m
cilium-d8grw                              1/1     Running            2 (18m ago)      174m
cilium-g4gmf                              1/1     Running            2 (18m ago)      174m
cilium-h5j4h                              1/1     Running            2 (18m ago)      174m
cilium-n62nv                              1/1     Running            2 (18m ago)      174m
cilium-operator-76cff99967-6fgkv          1/1     Running            2 (18m ago)      174m
cilium-operator-76cff99967-pbr4l          1/1     Running            2 (18m ago)      174m
cilium-w4n6d                              1/1     Running            2 (18m ago)      174m
cilium-wgm7l                              1/1     Running            2 (18m ago)      174m
cilium-zqb6w                              1/1     Running            2 (18m ago)      174m
coredns-d76bd69b-bhgnl                    0/1     CrashLoopBackOff   44 (3m27s ago)   177m
hubble-relay-67f64789c7-cjzz9             0/1     CrashLoopBackOff   63 (4m15s ago)   174m
hubble-ui-794cd44b77-9vgbl                3/3     Running            6 (18m ago)      174m
local-path-provisioner-6c79684f77-l4cqp   0/1     CrashLoopBackOff   35 (3m53s ago)   177m
metrics-server-7cd5fcb6b7-v74rc           0/1     CrashLoopBackOff   42 (3m35s ago)   177m

Any help is appreciated! thanks

Solution

  • Since you're using Cilium, I think you might be running into this issue: https://github.com/cilium/cilium/issues/10645

    The workaround is to ensure net.ipv4.conf.lxc*.rp_filter is set to 0:

    echo 'net.ipv4.conf.lxc*.rp_filter = 0' | sudo tee -a /etc/sysctl.d/90-override.conf
sudo systemctl start systemd-sysctl