Why is it that my Prometheus Operator ServiceMonitor needs a `release` label to run properly?

Question

When I deploy this:

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    app: complaints-monitor
    release: prometheus # <---------
  name: complaints-monitor-svc-monitor
  namespace: default
spec:
  endpoints:
    - path: /metrics
      port: web
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app: complaints-monitor

My service shows up under prometheus targets and runs correctly.

If I apply the following instead, the target disappears:

kind: ServiceMonitor
apiVersion: monitoring.coreos.com/v1
metadata:
  labels:
    app: complaints-monitor
  name: complaints-monitor-svc-monitor
  namespace: default
spec:
  endpoints:
    - path: /metrics
      port: web
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app: complaints-monitor

Answer 1

Just figured out this problem after checking the values.yaml of kube-prometheus-stack

• vaules.yaml

    ## If true, a nil or {} value for prometheus.prometheusSpec.serviceMonitorSelector will cause the
    ## prometheus resource to be created with selectors based on values in the helm deployment,
    ## which will also match the servicemonitors created
    ##
    serviceMonitorSelectorNilUsesHelmValues: true


    ## ServiceMonitors to be selected for target discovery.
    ## If {}, select all ServiceMonitors
    ##
    serviceMonitorSelector: {}
    ## Example which selects ServiceMonitors with label "prometheus" set to "somelabel"
    # serviceMonitorSelector:
    #   matchLabels:
    #     prometheus: somelabel

When you use helm to install kube-prometheus-stack, it adds label release: <prometheus-installed-namespace> to Kubernetes resource.

If setting serviceMonitorSelectorNilUsesHelmValues is true a selector will be add to prometheus.prometheusSpec.serviceMonitorSelector, in your case is release: prometheus.

You shall be able to see the selector for prometheus to select serviceMonitor with command

kubectl -n <prometheus-installed-namespace> get prometheus -o yaml

and search for serviceMonitorSelector.

You have two options to get it work without adding release label

1. Set serviceMonitorSelectorNilUsesHelmValues to false, then prometheus will select all serviceMonitor

2. Or, set serviceMonitorSelector to any label you like.