Search code examples
azurekubernetesprometheusprometheus-alertmanagerprometheus-node-exporter

2 different prometheus server to scrapping different data but shows combined in their /targets

I have 2 different prometheus server (with its set of alertmanager, pushgateway, nodeExporter and kube-state-metrics)

When I port forward either of these 2 servers, in their respective /targets - I find combined details for the below job_name(s) :

  1. kubernetes-nodes (20/20 up)
  2. kubernetes-nodes-cadvisor (20/20 up)
  3. kubernetes-pods (90/90 up)
  4. kubernetes-service-endpoints (78/78 up)
  5. prometheus-pushgateway (2/2 up)
  6. prometheus (1/1 up)
  7. kubernetes-apiservers (1/1 up)

My values.yaml file (different for both servers) contains the following part -

prometheus:
  serverFiles:
    prometheus.yml:
      rule_files:
        - /etc/config/recording_rules.yml
        - /etc/hcp/config/alerting_rules.yml
      scrape_configs:
        - job_name: prometheus
          static_configs:
            - targets:
                - localhost:9090 #exposing metrics

I want to ensure that for prometheus-server-1 should only scrape details related to it and same goes with prometheus-server-2, & have only their details in their /targets despite having the same default ports used by prometheus.

Solution

  • While going through all the prometheus docs and experimenting different ways, finally am able to get through to have prometheus-server scraping data only related to it, hence answering my own question.

    This can be achieved by adding relabel_configs condition in the respective jobs under values.yaml file shared above.

    Example for job_name: prometheus-pushgateway:

    prometheus:
  serverFiles:
    prometheus.yml:
      rule_files:
        - /etc/config/recording_rules.yml
        - /etc/hcp/config/alerting_rules.yml
      scrape_configs:
        - job_name: prometheus
          static_configs:
            - targets:
                - localhost:9090 #exposing metrics
      - job_name: prometheus-pushgateway
        kubernetes_sd_configs:
        - role: service
        relabel_configs:
        - action: keep
          regex: pushgateway
          source_labels:
          - __meta_kubernetes_service_annotation_prometheus_io_probe
        # Added below condition to ensure it keeps only specific pushgateway
        - action: keep
          regex: first-prometheus-pushgateway.namespace.svc:9091
          source_labels:
          - __address__

    **NOTE: scrape_configs overwrites all the default scrape_configs, so kindly include all default statements while adding your specific conditions to it.

    For additional info, below links can be helpful: