Scrape jenkins metrics with prometheus

Question

I'm new to Prometheus, so I'm not sure what I'm doing wrong but these are my service and service monitor definitions.

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8080'
    prometheus.io/path: '/prometheus'
  labels:
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/component: jenkins
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: jenkins
  labels:
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/component: jenkins
    release: prometheus
spec:
  endpoints:
  - interval: 10s
    path: /prometheus/
    port: "8080"
  jobLabel: app.kubernetes.io/instance
  selector:
    matchLabels:
      app.kubernetes.io/component: jenkins
      app.kubernetes.io/instance: jenkins

But my Jenkins doesn't appear under the targets list on the Prometheus UI. It appears under Service Discovery which makes me believe that the Operator is correctly picking it up through the release: prometheus label.

I have installed the prometheus plugin on jenkins and I'm able to view metrics when I curl https://<JENKINS_URL>/prometheus/

What I'm trying to figure out is why Jenkins doesn't appear under the targets list.

Is there any proper documentation on how to go about this or can anyone who has successfully implemented this share any pointers?

Answer 1

There is no better docs than reading the code itself.

You need pay attention to this line in the custom resource definition of ServiceMonitor.

port:
  description: Name of the service port this endpoint refers to.
               Mutually exclusive with targetPort.
  type: string

Basically, you created a serviceMonitor to a service port named "8080".

endpoints:
  - interval: 10s
    path: /prometheus/
    port: "8080"

But you defined an unnamed service whose port number is 8080.

spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080

Do you see the mismatch now?

You need either use targetPort: 8080 and targetPort only in serviceMonitor,

or, even better, use port: "web" in serviceMonitor, and at the same time, name your service "web".

ServiceMonitor:

endpoints:
  - interval: 10s
    path: /prometheus/
    port: "web"

Service:

spec:
  type: ClusterIP
  ports:
    - name: "web"
      port: 8080
      targetPort: 8080