Search code examples
monitoringprometheus

Is it possible to replicate Prometheus?

is there possibility to replication Prometheus?

For example, there are two instances of Prometheus. First of them is turned off and second one takes over his duties. Is it possible? I know there is Federation, but it this case second one takes samples only when the first one works. I don't want the second one to be the same instance as the first one, but a kind of replica.

Solution

  • You are asking about hot/warm architecture high availability for Prometheus.

    There are two aspects to your question:

    1. The storage of data: Prometheus has the ability of using remote storage that can be used to provide HA for your data
    2. The scraping of targets: there is no builtin mechanism to scrape in cluster.
    • Either both Prometheus scrape at the same time (but then you have hot/hot)
    • or you find a way to detect a Prometheus is down (prometheus scrape each other) and trigger an action (quite easy to do with a webhook) - by example enabling config and triggering the reload

    If HA is really important for you, you'd rather:

    • use a hot/hot setup with some proxy like Trikster or clustering like Thanos
    • switch to Cortex which is like a de-structured Prometheus to provide scalability and HA

    If you can afford some small downtime of Prometheus, you can also just let the scheduler re-schedule Prometheus and persist the data (maybe with remote read/write).