How to grok a pipe-delimited string in a log line

Question

I need to grok a pipe-delimited string of values in a grok line; for example:

|NAME=keith|DAY=wednesday|TIME=09:27:423227|DATE=08/06/2019|amount=68.23|currency=USD|etc...

What is the easiest way to do this?

Is there any form of a grok split?

Thanks, Keith

Answer 1

Your scenario is the perfect use case of logstashs kv (key-value) filter!

The basic idea behind this filter plugin is to extract key-value pairs in a repetitive pattern like yours.

In this case the field_split character would be the pipe ( | ).

To distinguish keys from values you would set the value_split character to the equal sign ( = ).

Here's a sample but untested filter configuration:

filter{
  kv{
    source => "your_field_name"
    target => "kv"
    field_split => "\|"
    value_split => "="
  }
}

Notice how the pipe character in the field_split setting is escaped. Since the pipe is a regex-recognized character, you have to escape it!

This filter will extract all found key-value pairs from your source field and set it into the target named "kv" (the name is arbitrary) from that you can access the fields.

You might want to take a look at the other possible settings of the kv filter to satisfy your needs.

I hope I could help you! :-)