Do metrics published by the Jaeger Operator include those from the Collector and Query services too?

Question

I am deploying Jaeger using the Jaeger Operator and it seems to be working fine. However, now I am trying to set up Prometheus metrics scraping (using the Prometheus Operator) but I am not seeing a Service in my cluster that exposes the metrics ports for the Jaeger Collector (port 14269) or Query services (port 16687) (port number reference from the Jeager Monitoring documentation).

The only relevant Service I see is jaeger-operator-metrics:

$ kubectl get svc -A
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)
simple-prod-collector           ClusterIP      10.43.20.131    <none>           9411/TCP,14250/TCP,14267/TCP,14268/TCP
simple-prod-query               ClusterIP      10.43.141.211   <none>           16686/TCP
simple-prod-collector-headless  ClusterIP      None            <none>           9411/TCP,14250/TCP,14267/TCP,14268/TCP
jaeger-operator-metrics         ClusterIP      10.43.90.169    <none>           8383/TCP,8686/TCP

I am able to set up a Prometheus ServiceMonitor to scrape metrics from this service but I am not sure if this includes the metrics that are normally gathered by the Collector and Query microservices or not... I am guessing not as that would seem to violate the premise of microservices.

Is there some setting in the Jaeger Operator spec that I missed for exposing those metrics endpoints in the other components?

Answer 1

I figured it out... the Jaeger Operator doesn't create a Service exposing the metrics endpoints. These endpoints are just exposed via the pods for the Collector and Query components.

An example from the Collector pod spec:

    ports:
    - containerPort: 9411
      name: zipkin
      protocol: TCP
    - containerPort: 14267
      name: c-tchan-trft
      protocol: TCP
    - containerPort: 14268
      name: c-binary-trft
      protocol: TCP
    - containerPort: 14269
      name: admin-http
      protocol: TCP
    - containerPort: 14250
      name: grpc
      protocol: TCP

Note the admin-http port there.

So to get the Prometheus Operator to scrape these metrics, I created a PodMonitor which covers both the Collector and Query components because both of them have the labels/app: jaeger and admin-http ports defined:

cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: jaeger-components
  namespace: monitoring
  labels:
    release: prometheus
spec:
  podMetricsEndpoints:
  - path: /metrics
    port: admin-http
  namespaceSelector:
    matchNames:
    - monitoring
  selector:
    matchLabels:
        app: jaeger
EOF