How can I plot the time taken on each API hit on a certain endpoint using prometheus?

Question

I'm trying to create some metrics in Prometheus. I want to measure how long does it take to return me the response when I hit some endpoint. So I've annotated the route with @Timed.

Here is my rest controller:

import io.micrometer.core.annotation.Timed;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class MyController {

    @GetMapping("/ping")
    @Timed(value = "ping.request.time", description = "time pIng")
    public String ping(){
        return "pIng";
    }

    @GetMapping("/pong")
    @Timed(value = "pong.request.time", description = "time pOng")
    public String pong(){
        return "pOng";
    }
}

But after this in http://localhost:8080/actuator/prometheus nothing appears mentioning ping or pong. But after hitting these API for at least one time, they appear. The response looks something like this:

...
# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 2.0
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.5708052
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/ping",} 1.0
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/ping",} 0.0085949
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics",} 1.0
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics",} 0.1374681
http_server_requests_seconds_count{error="none",exception="none",method="GET",outcome="CLIENT_ERROR",status="404",uri="/**",} 1.0
http_server_requests_seconds_sum{error="none",exception="none",method="GET",outcome="CLIENT_ERROR",status="404",uri="/**",} 0.0148576
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.0508929
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/ping",} 0.0085949
http_server_requests_seconds_max{error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics",} 0.0
...

But when I try to execute this query: rate(http_server_requests_seconds_sum{uri="/ping"}[5m])/rate(http_server_requests_seconds_count{uri="/ping"}[5m]), it gives me Empty query result

I have also tried other queries such as:

• http_server_requests_seconds_sum{uri="/ping"}
• http_server_requests_seconds_count{uri="/ping"} But the result is the same.

How can I plot time taken on each API hit on /ping using prometheus?

Here is my prometheus.yaml file:

global:
  scrape_interval:     15s # Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['SYSTEM_IP_ADDRESS:8080'] # refer system ip address rather that localhost

And here is how I ran my system docker run -d --name=prometheus -p 9090:9090 -v C:\Users\Admin\Desktop\prometheus-test\prometheus.yaml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

Answer 1

Verify that Prometheus actually gets showed in question metrics: if they were present you'd get some result with your query. You can verify their presence by query http_server_requests_seconds_sum{uri="/ping"} or just http_server_requests_seconds_sum.

If nothing returned - check targets page, to see if your target is reachable by Prometheus and is actually scraped.

If target is not reachable - recheck configuration. This line

  - targets: ['SYSTEM_IP_ADDRESS:8080'] # refer system ip address rather than localhost

If you are using docker use host.docker.internal instead.

From:

To this: