Search code examples
configurationmicroservicesprometheus

Prometheus collect metrics from different microservices (metrics_paths) but same target (ip and port)

I want to collect metrics from different microservices from application my_app.

I want to be able to group them, e.g. get sum of requests in all microservices on one environment or compare metrics from same microservice on different environments.

dev_env.com/api/a/actuator/prometheus
dev_env.com/api/b/actuator/prometheus
dev_env.com/api/c/actuator/prometheus
test1_env.com/api/a/actuator/prometheus
test1_env.com/api/b/actuator/prometheus
test1_env.com/api/c/actuator/prometheus

Currently I plan to configure prometheus with as many jobs as microservices like this:

- job_name: "dev_a"
  metrics_path: "/api/a/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "a"

- job_name: "dev_b"
  metrics_path: "/api/b/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "b"
...

Is there a way to solve this problem with only one job? Is it correct approach to have one or multiple jobs in such a scenario? In reality there are really a lot of microservices and few environments.

Goal: have metrics with fields env, microservice, app. Then i will be able to easily manipulate with them. If there would be solution with one job, then i don't need app field, because job field would replace it. Now i need app to distinguish metrics from my_app and other metrics, e.g. kafka, servers.

I saw similar problem on stackoverflow but the difference here is I have the same ip and port for all microservices. Also I didn't find easy way for specifying multiple metrics_paths for one job.

Solution

  • I think you can try to create a dynamic metrics_path:

    - job_name: "foo"
  relabel_configs:
  # this relabeling puts "microservice" label value in place of "$1" in /api/$1/actuator/prometheus
  - source_labels: [microservice]
    target_label: __metrics_path__
    regex: (.+)
    replacement: /api/$1/actuator/prometheus
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "a"
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "b"

    It is possible to move further and avoid repeating the same host, but the configuration will be much more difficult to understand. Nevertheless, here comes the real relabeling kung-fu:

    # The idea behind this is to iterate not hosts, but different paths.
# To do so, we're going to write "microservice" label values into the array of targets,
# while the real target is going to be set with the "host" label.
- job_name: wicked
  static_configs:
    - targets: [a,b,c]
      labels:
        host: localhost
        app: my_app
        env: dev
  relabel_configs:
  # First, we're going to save target as microservice label
  - source_labels: [__address__]
    target_label: microservice
  # Then construct the metrics_path, just as before
  - source_labels: [microservice]
    target_label: __metrics_path__
    regex: (.+)
    replacement: /api/$1/actuator/prometheus
  # Now we need to set the real address of the server instead of microservice name
  - source_labels: [host]
    target_label: __address__
  # Finally, get rid of the host label
  - regex: ^host$
    action: labeldrop

    And the result is: result