prometheus not able to scrape uwsgi

Question

I am trying prometheus to scrape uwsgi container on port 7070

I have following scrape job in prometheus.yaml

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2

Also have annotations for uwsgi container :

  annotations:
     prometheus.io/scrape: "true"
     prometheus.io/port: "7070"

If I curl http://<host_ip>:7070/metrics from any container, it return json format metrics.

But when I do curl http://<host_ip>:7070/metrics | promtool check metrics.

I get error while linting: text format parsing error in line 1: invalid metric name

I think this is because uwsgi container expose metrics in json format and prometheus doesn't understand json.

What I need to do to make it scrapeable by prometheus ?

Answer 1

Manage to do this with https://github.com/timonwong/uwsgi_exporter as a sidecar container.

  - name: uwsgi
    image: .....
  - name: uwsgi-exporter
    image: timonwong/uwsgi-exporter:latest
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
    ports:
    - name: uwsgi-exp-port
      containerPort: 9117
      protocol: TCP
    args:
      - --stats.uri=http://localhost:7070/metrics