Search code examples
jsongrafanagrafana-lokilogql

Regexp JSON filtering in LogQL

I'd like to translate Kibana query like to following to LogQL:

host:("test1-myservice-*") AND level:ERROR 
AND NOT logger_name:"com.example.ExampleClass" 
AND _exists_:stack_trace 
AND NOT stack_trace:(
    "interrupted"
    OR "Read timed out"
    OR "java.lang.InterruptedException"
)

I have tried the following in Grafana Explore but it does not return any records for our JSON log messages:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace =~ ".*InterruptedException.*"

While using != instead of =~ it returns all records:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ ".*InterruptedException.*"

If I'm right the following applies from the documentations to the stack_trace field of the JSON log line:

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

Source: Label filter expression

The following seems to work but it seems awkward:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
!~ ".*InterruptedException.*|.*Read timed out.*"
| json

Furthermore, if I'm right, it searches for InterruptedException and Read timed out substrings in the complete JSON string instead of its stack_trace field only.

Is there a more LogQL-ish way to translate the Kibana query above to LogQL? Should the !~ operator work in this case?

Environment: Grafana 7.5.4 / 8.2.3, Loki: 2.4.1

Solution

  • This works:

    {host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ "(?s).*InterruptedException.*"

    Note the (?s) which enables matching new lines for the regex . character. (The stack_trace field of the JSON log message usually contain multiple lines.)

    This is also mentioned in the Log stream selector part of the documentation:

    Note: The =~ regex operator is fully anchored, meaning regex must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term\n.