Unable to run Prometheus in Portainer stack

Question

I am working with a Docker compose file taken from this Github repo, and it works well locally.

I then deployed the Docker compose on an Ubuntu 22.04 VM using Portainer. Now the Prometheus container keeps on restarting. There are two other containers, Grafana and an app I created, and they're fine.

Here's the error message:

caller=main.go:479 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="open /etc/prometheus/prometheus.yml: no such file or directory"

Here's the Docker compose file, which was added to create a Stack in Portainer :

version: "3.8"

volumes:
  prometheus_data: {}
  grafana_data: {}

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    volumes:
      - ./prometheus:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
    ports:
      - 9090:9090

  grafana:
    image: grafana/grafana
    container_name: grafana
    user: "472"
    restart: always
    environment:
      GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    env_file:
      - stack.env
    ports:
      - 3000:3000
    depends_on:
      - prometheus

  myapp:
    image: ...

The file etc/prometheus/prometheus.yml exists in the Docker container running locally on my machine. Since it is the same Docker compose file with the exact same commands, I don't understand why the file is not found.

Should this prometheus.yml file be mounted another way?

I am lost and don't know how to resolve this. Any help is welcome.

EDIT: Before, I was adding the Docker compose file manually by copy-pasting the file. Now I created a Github repo dedicated to this stack, so that it has access to the prometheus.yml file. Unfortunately, there's still the same problem.

Answer 1

What the problem was :

Portainer had no access to ./prometheus, and that's why it was crashing. When removing this line in the Docker file as well as the command section, it wasn't crashing anymore (logic right). It created automatically a prometheus.yml file, with Prometheus as the only target.

How to resolve this :

I had to create the prometheus.yml file in the Virtual Machine via SSH, and then mount it to the container.

Here's what the Docker compose file looks like now:

version: "3.8"

volumes:
  prometheus_data: {}
  grafana_data: {}

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    volumes:
      - /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
      
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
    ports:
      - 9090:9090

  grafana:
    image: grafana/grafana
    container_name: grafana
    user: "472"
    restart: always
    environment:
      GF_INSTALL_PLUGINS: "grafana-clock-panel,grafana-simple-json-datasource"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    env_file:
      - stack.env
    ports:
      - 3000:3000
    depends_on:
      - prometheus

  pax-exporter:
    image: ...
    container_name: pax-exporter
    restart: always
    env_file:
      - stack.env
    ports:
      - 8001:8000

There's apparently also another way to do it if you have Docker swarm. You can directly upload files into a volume. Here's the link to the documentation.

At least I learned something! Hope this can help someone