Search code examples
elasticsearchlogstashkibanalogstash-groklogstash-configuration

I can not read split fields in Logstash

I can distinguish the "msg" field in logstash in the following format

filter {
   kv {
     field_split => "|"
         source => "msg"
   }
 }

Properly seperated.

img1

But then the reserved area "latitude" is not processed

img 2

Adding as string

     " deviceValue" => "null ",
      **"test1" => "%{latitude}"**,
       " timeLabel" => "NOON ",
" appllicationName" => "null ",
       " longitude" => "29.08222 ",

Thank you for your help

Solution

  • Take a closer look to the parsed values. I believe they are not in fact properly separated. You have spaces in the source data surrounding your split character "|", so when it is parsed you actually don't get a field named "latitude" but " latitude". From your post:

    " longitude" => "29.08222 ",

    Do you see the leading space on " longitude" and the trailing one in the value?

    I assume you don't need those, so one way to resolve the problem would be to clean the whitespace from the source data and then use your existing scripts.
    Alternatively, if you cannot modify the source data, you can set your filter to split on " | ":

    filter {
   kv {
     field_split => " | "
         source => "msg"
   }
 }

    And finally, if you indeed need those spaces and cannot change that, you can change "%{latitude}" to "%{ latitude}".