Prometheus receiver in OTEL collector is not reading/processing scrape config

Question

I am trying to configure my Otel collector to only pull metrics from pods that have annotation, prometheus.io/scrape=true, but metrics are being scraped from pod where the annotation is set as prometheus.io/scrape=false. I am struggling to understand what is wrong in the configuration below. There is no information in the OTEL collector logs (set to debug).

receivers:
        prometheus:
          config:
            scrape_configs:
              - job_name: 'otel-collector-k8s-pods'
                scrape_interval: 30s
                scrape_timeout: 2s
                kubernetes_sd_configs:
                - role: pod
                tls_config:
                  insecure_skip_verify: true
                relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                  action: keep
                  regex: true
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
                  action: replace
                  target_label: __scheme__
                  regex: (https?)
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                  action: replace
                  target_label: __metrics_path__
                  regex: (.+)
                - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                  action: replace
                  target_label: __address__
                  regex: (.+)(?::\d+);(\d+)
                  replacement: $1:$2
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                - source_labels: [__meta_kubernetes_namespace]
                  action: replace
                  target_label: namespace
                - source_labels: [__meta_kubernetes_host_name]
                  action: replace
                  target_label: kubernetes_pod_name

Answer 1

I've set a simple test, one pod running with annotation prometheus.io/scrape: 'true' and one with prometheus.io/scrape: 'false'

Metrics are only read from the pod having scrape set to true

The moment I've set scrape to true for the second pod, metrics started coming in which means annotations are working.

I'm using a simpler config. Maybe in the process of relabeling something is being messed in your config.

Try adding the relabel configs one by one to see which one's the culprit. Here is the working config:

    prometheus:
      config:
        scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 10s
            static_configs:
              - targets:
                  - ${MY_POD_IP}:8888
          - job_name: k8s
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: "true"
              action: keep

P.S. using opentelemetry-collector-0.30.0 helm chart here