Using namespaceSelector in ServiceMonitor Kube-prometheus

Question

I'm studying Prometheus Operator from the project: https://github.com/prometheus-operator/kube-prometheus

When my service/deployment is in the same "monitoring" namespace as prometheus I have success to visualize metrics. Example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: caixa-app
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-caixa
    app.kubernetes.io/part-of: kube-prometheus
spec:
  endpoints:
    - interval: 15s
      port: http
  selector:
    matchLabels:
      app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: caixa-deployment
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: caixa-pod
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "80"
      labels:
        app: caixa-pod
    spec:
      containers:
      - name: caixa
        image: 'REGISTRY-NAME/caixa:1.0'
        ports:
          - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
      imagePullSecrets:
      - name: regcred      
---
apiVersion: v1
kind: Service
metadata:
  name: caixa-svc
  namespace: monitoring
  labels:
    app: caixa
spec:
  selector:
    app: caixa-pod
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: ClusterIP

In the way mentioned above I have success in the same namespace "monitoring"

However I need my Prometheus-operator to collect metrics from other namespaces that I will create. So I used the parameter "namespaceSelector".

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: caixa-app
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-caixa
    app.kubernetes.io/part-of: kube-prometheus
spec:
  endpoints:
    - interval: 15s
      port: http
  namespaceSelector:
    matchNames:
    - shop
  selector:
    matchLabels:
      app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: caixa-deployment
  namespace: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: caixa-pod
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "80"
      labels:
        app: caixa-pod
    spec:
      containers:
      - name: caixa
        image: 'REGISTRY-NAME/caixa:1.0'
        ports:
          - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
      imagePullSecrets:
      - name: regcred      
---
apiVersion: v1
kind: Service
metadata:
  name: caixa-svc
  namespace: shop
  labels:
    app: caixa
spec:
  selector:
    app: caixa-pod
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: ClusterIP

That way the metrics don't appear in prometheus, but they are in my Prometheus config.

- job_name: serviceMonitor/monitoring/caixa-app/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
....
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - shop

Do I need to add any more elements? so that Service Monitor can collect metrics from other namespaces?

I already validated in my ClusterRole and Prometheus Operator has all permissions in namespaces.

Answer 1

Problem solved!

In Kube-prometheus by default it maps namespaces: (default, kube-system, monitoring)

To add specific namespaces we must edit the two files: prometheus-roleSpecificNamespaces.yaml

- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.38.0
    name: prometheus-k8s
    namespace: loja
  rules:
  - apiGroups:
    - ""
    resources:
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - extensions
    resources:
    - ingresses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - networking.k8s.io
    resources:
    - ingresses
    verbs:
    - get
    - list
    - watch- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.38.0
    name: prometheus-k8s
    namespace: loja
  rules:
  - apiGroups:
    - ""
    resources:
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - extensions
    resources:
    - ingresses
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - networking.k8s.io
    resources:
    - ingresses
    verbs:
    - get
    - list
    - watch

prometheus-roleBindingSpecificNamespaces.yaml

- apiVersion: rbac.authorization.k8s.io/v1
  kind: RoleBinding
  metadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.38.0
    name: prometheus-k8s
    namespace: loja
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: Role
    name: prometheus-k8s
  subjects:
  - kind: ServiceAccount
    name: prometheus-k8s
    namespace: monitoring

After adding the desired namespace and applying the yml, the ServiceMonitor will be added.