Search code examples
devopsprometheusgrafanametricspromql

How can I take all targets' metrics in one page at Prometheus


I have Prometheus setup on AWS EC2. I have 11 targets configured which have 2+ endpoints. I would like to setup a endpoint/query etc to gather all the metrics in one page. I am pretty stuck right now. I could use some help thanks:)

my prometheus targets file


Solution

    1. Prometheus adds an unique instance label per each scraped target according to these docs.
    2. Prometheus provides an ability to select time series matching the given series selector. For example, the following series selector selects time series containing {instance="1.2.3.4:56"} label, e.g. all the time series obtained from the target with the given instance label.
    3. Prometheus provides the /api/v1/series endpoint, which returns time series matching the provided match[] series selector.

    So, if you need obtaining all the time series from a particular target my-target, you can issue the following request to /api/v1/series:

    curl 'http://prometheus:9090/api/v1/series?match[]={instance="my-target"}'
    

    If you need obtaining metrics from the my-target at the given timestamp, then issue the query with the series selector to /api/v1/query:

    curl 'http://prometheus:9090/api/v1/query?query={instance="my-target"}&time=needed-timestamp'
    

    If you need obtaining all the raw samples from the my-target on the given time range (end_timestamp+d ... end_timestamp], then use the following query:

    curl 'http://prometheus:9090/api/v1/query?query={instance="my-target"}[d]&time=end_timestamp'
    

    See these docs for details on how to read raw samples from Prometheus.

    If you need obtaining all the metrics / series from all the targets, then just use the following series selector: {__name__!=""}

    See also /api/v1/query_range - this endpoint is used by Grafana for building graphs from Prometheus data.