Search code examples
kubernetesflannel

Can not access Cluster Ip with same Node

I'm Deploying a k8s cluster on CentOS with version of v1.5.1

Having three nodes:

  • kube-01(master)
  • kube-02
  • kube-03

Having a deployment with one pod, named Deployment-A with a pod ip Pod-A-IP, deployed on kube-03

Having a deployment with two pods, named Deployment-B. Each work node has one pod. We call Pod-B-02 on kube-02 and Pod-B-03 on kube-03

Exposing Deployment-A using type NodePort, I have a Cluster IP Service-A-IP

  • Pod-B-02 access Service-A-IP, OK
  • Pod-B-03 access Service-A-IP, time out
  • kube-02 access Service-A-IP, OK
  • kube-03 access Service-A-IP, OK

It seems that accessing a Service from pods in same Node with the Service's backend will encounter this problem.

Updated on Mon Feb 20 16:22:47 CST 2017

I have captured the network traffic on Pod-B-03

  • 10.244.1.10 is the pod ip of Pod-B-03
  • 10.107.25.245 is Service-A-IP
  • 10.244.1.2 is Pod-A-IP

I'm using flannel. I suggested there is something wrong with flannel?

Solution

  • The problem you describe is something that I had in the past if I remember correct...but I had many network problems with many different error sources. If it is really the same issue, then probably setting net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to 1 may help. You could try this on all hosts first:

    sysctl -w net.bridge.bridge-nf-call-iptables=1
sysctl -w net.bridge.bridge-nf-call-ip6tables=1

    Then check the service networking again without rebooting your machine. If this helps, persist the change into /etc/sysctl.conf or /etc/sysctl.d/

    Kubernetes network problems tend to have uncountable sources of errors, making it really hard to debug this without enough information. It would be good if you could provide some additional information about how you did setup the cluster (kube-up, kargo, kops, kubeadm, ...), which cloud you use (or bare-metal?) and which network solution you chose (weave, calico, cloud provider based, ...)

    It may also help to see the output of iptables -L -t nat and the kube-proxy logs as most service related problems can be debugged with this information.

    EDIT I've just found the Kubernetes issue where I had this solution from: https://github.com/kubernetes/kubernetes/issues/33798