I have a kubernetes service set to externalTrafficPolicy: Cluster (it's a simple nginx backend). When i try to curl it from outside the cluster it's often timing out. The loadBalancerSourceRanges are set to 0.0.0.0/0, and it actually succeeds very infrequently (2/20 times).
I am aware that in an externalTrafficPolicy:Cluster service, the nodes in the cluster use iptables to reach the pod. So i did some tcpdumps from both the pod and a node in the cluster that is attempting to reach the pod
Below is a tcpdump from a node that the backend pod tried to reach and send data to. (note I am using Calico for my cluster CNI plugin). 10.2.243.236 is the IP of the backend pod
sudo tshark -i vxlan.calico | grep 10.2.243.236
Running as user "root" and group "root". This could be dangerous.
Capturing on 'vxlan.calico'
468 463 5.867497471 10.2.0.192 -> 10.2.243.236 TCP 58 38109 > http [SYN] Seq=0 Win=7514 Len=0 MSS=1460
464 5.867920115 10.2.243.236 -> 10.2.0.192 TCP 58 http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
604 599 7.372050068 10.2.243.236 -> 10.2.0.192 TCP 58 [TCP Retransmission] http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
759 781 9.372058511 10.2.243.236 -> 10.2.0.192 TCP 58 [TCP Retransmission] http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
1094 1078 13.372017415 10.2.243.236 -> 10.2.0.192 TCP 58 [TCP Retransmission] http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
1877 1913 21.372786131 10.2.243.236 -> 10.2.0.192 TCP 58 [TCP Retransmission] http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
3285 3281 37.372007425 10.2.243.236 -> 10.2.0.192 TCP 58 [TCP Retransmission] http > 38109 [SYN, ACK] Seq=0 Ack=1 Win=26200 Len=0 MSS=1310
So it basically seems like the node is initiating the TCP connection, but is not responding to the pod's syn-ack message, and eventually the connection times out.
How can i debug this further? I'm kind of stuck on how i can debug why the node is seemingly not responding to the connection it initiated in the first place.
NOTE: I can curl the pod IP successfully from inside every node.
Answer: We installed Calico on the kubernetes cluster as the CNI plugin. We did not set the kube proxy's --cluster-cidr argument as we believed Calico would take care of creating the rules.
Upon running iptables-save on kubernetes nodes, it was found that no rule actually matched the pod cidr range, and hence packets were getting dropped by the default FORWARD DROP rule (this can be verified using iptables-save -c).
after setting kube-proxy's cluster-cidr argument, and restarting kube proxy on all the worker nodes, the IPtables rules were created as expected and services with externalTrafficPolicy: Cluster worked as expected.