How can I measure each API response duration using `http_server_requests_seconds_sum`

Question

In Grafana I want to plot in how long my APIs are taking to return response.

So what I did was create a dashboard, then panel, and there I added these prometheus queries:

• http_server_requests_seconds_sum{uri="/doc"}
• http_server_requests_seconds_sum{uri="/list-doc"}
• http_server_requests_seconds_sum{uri="/cont"}
• http_server_requests_seconds_sum{uri="/list-cont"}
• ... // yes I need to work on API naming as well

Then it gave me a beautiful graph, like this:

I think this graph is actually plotting the summation of the duration, but I want to plot individual values. I can do the opposite, using reduce transformation. But I'm stuck here, your help will be really appreciated.

Detailed explaination:

Let's say I''ve made 4 requests (for the same endpoint)

• request 1 takes 1 sec,
• request 2 takes 2 sec,
• request 3 takes 1 sec,
• request 4 takes 4 sec,

So value http_server_requests_seconds_sum for this endpoint will become:

• 1
• 3 (1+2)
• 4 (3+1)
• 8 (4+4)

What I'm plotting right now is 1,3,4,8

What I want to plot is 1,2,1,4

How can I do so using this http_server_requests_seconds_sum??

I don't want to show average, or rate, I want to render specific duration of each API hit.

Answer 1

First we have to create a new Gauge and register it, like this:

    private final CollectorRegistry registry = CollectorRegistry.defaultRegistry;

    private final Gauge dictOfMetrics = Gauge.build()
            .name("duration_metric")
            .labelNames("uri")
            .help("duration_metric_help")
            .register(registry);

And then we can use a @Component class implementing HandlerInterceptor, and it has to steps:

• when the request lands on our server, we will add a new header stating the current time
• when the request is serrved, we will get that header, measure time difference, set it to gauge, and remove that header

Can be done like this:

@Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        request.setAttribute("x-start-time", System.currentTimeMillis());
        return true;
    }

@Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
        Long startTime = (Long) request.getAttribute("x-start-time");
        long executeTime = System.currentTimeMillis() - startTime;
        String uri = request.getRequestURI();
        request.removeAttribute("x-start-time");

        dictOfMetrics.labels(uri).set(Double.parseDouble(durationMapString));
    }