Add 2 fields in LogQL to use within aggregate function

Question

I have log lines that contain a few timestamp fields. Here is an example of log line I am filtering in order to process it:

    
{
  "time": "2022-06-22T10:33:08.710037238Z",
  "@version": "1",
  "message": "Duration at response processing ends",
  "logger_name": "com.mks.cloud.filters.MainFilter",
  "thread_name": "reactor-http-epoll-1",
  "level": "INFO",
  "level_value": 20000,
  "rqst_id_ts": "b65c37d9284584e71b1dcd84b6a74075",
  "rqst_end_ts": "1655893988698",
  "rqst_start_ts": "1655893988698",
  "rsp_start_ts": "1655893988709",
  "rsp_end_ts": "1655893988709"
}

What I would like to do is calculate a value that would represent the duration between 2 timestamps in the log line so that I can then put the obtained range into quantile_over_time() or any other aggregate function.

For instance, the following works:

quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
      | json
      | __error__ = ""
      | unwrap rsp_end_ts | __error__="" [5m]) by (tsNs)

However this is not what I want to do since calculating the quantile of epoch timestamps makes no sense. What I want to calculate is the p99 of (rsp_end_ts - rqst_start_ts).

I tried the following which of course doesn't work, but gives an idea as to what I am attempting to do:

quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
      | json
      | __error__ = ""
      | unwrap (rsp_end_ts - rqst_start_ts) | __error__="" [5m]) by (tsNs)

If somehow there was a way to create a new label like rqst_duration=(rsp_end_ts - rqst_start_ts)

Then the following would be what I am looking for:

quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
      | json
      | __error__ = ""
      | unwrap rqst_duration | __error__="" [5m]) by (tsNs)

I couldn't find any documentation about this, which is very surprising, I would think (but it seems I might be wrong) this to be a common use case. Any help would be greatly appreciated :).

Answer 1

You can use template functions for that. Here's a sample query on the Grafana playground that you can inspire from.

So your query will look something like:

{app="myapp"} |~ "rsp_end_ts"
      | json
      } label_format result=`{{sub .rsp_end_ts rqst_start_ts}}` | line_format "{{ .result }}" 

Then you can use the result.