Prometheus error: couldn't load configuration. Found multiple scrape configs

As the title says, I have ran into a prometheus exception and cannot figure out how to fix it.

We use a BOSH deployment like so:

bosh -d prometheus deploy -n prometheus.yml \
-o prometheus-boshrelease/manifests/operators/monitor-concourse.yml \
-o prometheus-boshrelease/manifests/operators/monitor-bosh.yml \
...

We then do a manifest update like so:

- type: replace
path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/scrape_configs/-
value:
  job_name: concourse
  scrape_interval: 2m
  scrape_timeout: 1m
  static_configs:
    - targets:
      - 192.168.123.123:9391

During the deploy we get an error that the prometheus2 instance is not running. Within the prometheus.stderr.log is where I found the error message:

found multiple scrape configs with job name \"concourse\"

How can I get the bosh and concourse exporters to work together? Is it something to do with my manifest modification? Looking for any ideas, thank you!

Solution

You already add this scrape config by applying the ops-file monitor-bosh.yml. https://github.com/bosh-prometheus/prometheus-boshrelease/blob/master/manifests/operators/monitor-bosh.yml#L199-L219

This enables auto discovery of your concourse deployment (if concourse is also deployed by bosh).

If you do not want to use auto-discovery but staticly configure your concourse endpoint (what I do not recommend though), you have to remove the part I linked from your deployment first. This could be done with an ops-file like this (not tested):

- type: remove
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/scrape_configs/job_name=concourse?

It has to be applied before you apply your changes.