Search code examples
grafanagrafana-lokilogql

How to sort 500-511 HTTP errors in a file produced daily on Grafana using Loki as Datasource

How to grep for only 500 errors (500-511) in a file that is created daily with date stamp in file name. These files are configured to be pushed by promtail agent to Loki server so I can visualize them on Grafana. The files are being produced daily and this is an example of the file sss2022-03-16.txt. I would like to count and visualise this on grafana only the 500-511 HTTP errors for file produced each day.

On Grafana Loki I tried doing this {job="cbas-dev-logs"} |= "500|501|502|503|504|505|506|507|508|509|510|511" but that didn't grep specifically only the 500s HTTPs

Below you can see an example of the file sss2022-03-16.txt

10.32.10.181 ignore 19 Feb 2022 00:26:04 GMT 10.32.10.44 GET / HTTP/1.1 500 73 N 0 h 10.32.26.124 ignore 19 Feb 2022 00:26:06 GMT 10.32.10.44 GET / HTTP/1.1 501 73 N 0 h 10.32.42.249 ignore 19 Feb 2022 00:26:27 GMT 10.32.10.44 GET / HTTP/1.1 500 73 N 1 h 10.32.10.181 ignore 19 Feb 2022 00:26:34 GMT 10.32.10.44 GET / HTTP/1.1 302 73 N 0 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 503 73 N 1 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 502 73 N 1 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 502 73 N 1 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 504 73 N 1 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 511 73 N 1 h 10.32.26.124 ignore 19 Feb 2022 00:26:36 GMT 10.32.10.44 GET / HTTP/1.1 508 73

Solution

  • your query selects not only HTTP statuses 5XX but all lines that contains 500-511 number. Try one of the solutions below:

    Solution-1: try to use more specific query like this one :

    {job="cbas-dev-logs"} |~ "HTTP/1.1 (500|501|502|503|504|505|506|507|508|509|510|511)"

    Update: I have tried and it works for me. Example for 500-511 codes: enter image description here

    Solution-2: use patter parser, so your query will look:

    {job="cbas-dev-logs"} | pattern "<_> <_> <_> <_> <_> <_> <_> <_> <_> <_> <_> <status> <_>" | status >= 500 and status <= 511

    here is an example of the results:

    Moreover, using pattern parser you can use status in the aggregation functions.

    enter image description here