Prometheus failing to pick up formatted metrics that service discovery can find

Question

I installed kube-prometheus-stack via helm:

kubectl create namespace monitoring && \
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \
helm repo update && \
helm install -n monitoring prometheus-stack prometheus-community/kube-prometheus-stack

Then proceeded to deploy a fastapi application which has a metrics endpoint that Prometheus is supposed to scrape. /metrics endpoint works fine as seen below:

from starlette_prometheus import metrics, PrometheusMiddleware
from fastapi import FastAPI, Request


app = FastAPI()

# Add Prometheus metrics as middleware
app.add_middleware(PrometheusMiddleware)
app.add_route("/metrics", metrics)

I expected both the target and service discovery to work but only service discovery appears to be working

• Prometheus Operator version:

  2.36.1

• Kubernetes version information:

  Client Version: v1.24.2

• Kubernetes cluster kind:

  Minikube v1.25.2-1

Here are the manifests for the deployed application:

1. Namespace and deployment

apiVersion: v1
kind: Namespace
metadata:
  name: code-detector-demo

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: code-detector
  namespace: code-detector-demo
  labels:
    release: prometheus-stack
spec:
  replicas: 1
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack
  template:
    metadata:
      labels:
        app: code-detector
        release: prometheus-stack
    spec:
      containers:
      - name: code-detector
        image: <MY-IMAGE:TAG>
        resources:
          limits:
            memory: 512Mi
            cpu: 1000m
        ports:
        - containerPort: 8000

2. Service

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
  labels:
    release: prometheus-stack
  name: code-detector-service
  namespace: code-detector-demo
spec:
  selector:
    app: code-detector
    release: prometheus-stack
  ports:
    - name: code-detector-port
      port: 8000
      targetPort: 8000

3. ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: code-detector-servicemonitor
  # same namespace that Prometheus is running in
  namespace: monitoring
  labels:
    app: code-detector
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack
  endpoints:
    - path: /metrics
      port: code-detector-port
      interval: 15s
  namespaceSelector:
    matchNames:
    - code-detector-demo # namespace where the app is running

Here's the labels in service discovery:

Targets pane don't show any picked up scraping job:

What could I be doing wrong?

Answer 1

You have in your servicemonitor:

spec:
  selector:
    matchLabels:
      app: code-detector
      release: prometheus-stack

which basicaly means "get all endpoints from services with labels app=code-detector and release=prometheus-stack".

Your service don't have the label app=code-detector.