Is there any way to know which pod the service is load-balanced in Kubernetes?

Question

I manage 3 Pods through Deployment and connect through NodePort of Service.
I wonder which pod the service load balanced whenever I connect from outside.
It's hard to check with Pods log, can I find out through the event or kubectl command?

Answer 1

I am not sure if this is exactly what you're looking for, but you can use Istio to generate detailed telemetry for all service communications.

You may be particularly interested in Distributed tracing:

Istio generates distributed trace spans for each service, providing operators with a detailed understanding of call flows and service dependencies within a mesh.

By using distributed tracing, you are able to monitor every requests as they flow through a mesh.
More information about Distributed Tracing with Istio can be found in the FAQ on Distributed Tracing documentation.
Istio supports multiple tracing backends (e.g. Jaeger).

Jaeger is a distributed tracing system similar to OpenZipkin and as we can find in the jaegertracing documentation:

It is used for monitoring and troubleshooting microservices-based distributed systems, including:

• Distributed context propagation
• Distributed transaction monitoring
• Root cause analysis
• Service dependency analysis
• Performance / latency optimization

Of course, you don't need to install Istio to use Jaeger, but you'll have to instrument your application so that trace data from different parts of the stack are sent to Jaeger.

---

I'll show you how you can use Jaeger to monitor a sample request.

Suppose I have an app-1 Deployment with three Pods exposed using the NodePort service.

$ kubectl get pod,deploy,svc
NAME                                        READY   STATUS    RESTARTS   AGE   IP           
app-1-7ddf4f77c6-g682z                      2/2     Running   0          25m   10.60.1.11   
app-1-7ddf4f77c6-smlcr                      2/2     Running   0          25m   10.60.0.7    
app-1-7ddf4f77c6-zn7kh                      2/2     Running   0          25m   10.60.2.5    

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-1                      3/3     3            3           21m

NAME                                         TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                      AGE
service/app-1                                NodePort       10.64.0.88    <none>           80:30881/TCP                 25m

Additionally, I deployed jaeger (with istio):

$ kubectl get deploy -n istio-system | grep jaeger
jaeger                 1/1     1            1           67m

To check if Jaeger is working as expected, I will try to connect to this app-1 application from outside the cluster (using the NodePort service):

$ curl <PUBLIC_IP>:30881
app-1

Let's find this trace with Jaeger:

As you can see, we can easily find out which Pod has received our request.