How to configure prometheus with alertmanager?

Question

docker-compose.yml: This is the docker-compose to run the prometheus, node-exporter and alert-manager service. All the services are running great. Even the health status in target menu of prometheus shows ok.

version: '2'

services:

    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanger/alert.rules:/alert.rules
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'

    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
             - ./alertmanager/alertmanager.yml:/alertmanager.yml
        command:
            - '--config.file=/alertmanager.yml'
        ports:
            - '9093:9093'

prometheus.yml

This is the prometheus config file with targets and alerts target sets. The alertmanager target url is working fine.

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

# this is where I have simple alert rules
rule_files:
  - ./alertmanager/alert.rules

scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
        - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['some-ip:9093']

alert.rules: Just a simple alert rules to show alert when service is down

ALERT service_down
  IF up == 0

alertmanager.yml

This is to send the message on slack when alerting occurs.

global:
  slack_api_url: 'https://api.slack.com/apps/A90S3Q753'

    route:
      receiver: 'slack'

    receivers:
      - name: 'slack'
        slack_configs:
          - send_resolved: true
            username: 'tara gurung'
            channel: '#general'
            api_url: 'https://hooks.slack.com/services/T52GRFN3F/B90NMV1U2/QKj1pZu3ZVY0QONyI5sfsdf'

Problems: All the containers are working fine I am not able to figure out the exact problem.What am I really missing. Checking the alerts in prometheus shows.

Alerts No alerting rules defined

Answer 1

Your ./alertmanager/alert.rules file is not included in your docker config, so it is not available in the container. You need to add it to the prometheus service:

prometheus:
    image: prom/prometheus
    privileged: true
    volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml
        - ./alertmanager/alert.rules:/alertmanager/alert.rules
    command:
        - '--config.file=/etc/prometheus/prometheus.yml'

    ports:
        - '9090:9090'

And probably give an absolute path inside prometheus.yml:

rule_files:
- "/alertmanager/alert.rules"

You also need to make sure you alerting rules are valid. Please see the prometheus docs for details and examples. You alert.rules file should look something like this:

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m

Once you have multiple files, it may be better to add the entire directory as a volume rather than individual files.

Moreover, you have not specified the correct hostname to the alertmanager in your prometheus.yml file. If you are using the shown docker compose configuration it should be:

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - 'alertmanager:9093'

Based on:

alertmanager:
    image: prom/alertmanager
    privileged: true
    volumes:
         - ./alertmanager/alertmanager.yml:/alertmanager.yml
    command:
        - '--config.file=/alertmanager.yml'
    ports:
        - '9093:9093'