I have k8s cluster (1.16) with one service in it's specific namespace, and a stateful set of rabbitmq (image: 3.8.0-alpine) with 3 replicas, one on each of my workers, in a different namespace.
I configured the service to connect rabbitmq with amqp://user:[email protected]:5672
I'm getting this error in the logs of my service:
Rabbit mq disconnected connect EHOSTUNREACH <rabbitmq_svc_ip>:5672
rabbitmq seems to be working fine. there are no errors shown in the pods.
What can I check to find the issue and make them communicate?
You asked for how to approach this issue, and so I am including how I would approach it.
I would describe your service to see if your pod ips are listed as endpoints behind it.
kubectl describe service <rabbitmq-service>
If they are not listed, then I would look at your service's selector to make sure that it matches your statefulset. If they are listed, then I would double check that your Service Port is actually 5672, and then I would exec into your pod where your app is running and try to ping the service name in your connection string to make sure it's reachable from your app's namespace. I would also try to do telnet tests to your service name and port to ensure that your app can connect to your service on that port, and I would telnet to your rabbitmq pod ips and ports to ensure that your app can connect to them. If telnet isn't in your image, I would install it for troubleshooting purposes.
kubectl exec -it <app-pod> -n <namespace> /bin/bash
telnet <pod ip or service name> <port>