After adding Prune filter along with KV filter - logs are not going to Elastic search

Question

I am learning ELK and trying to do as a POC for my project. I am applying KV filter for the sample integration logs from my project and i could see lot of extra fields are coming as a result so i have tried to apply prune filter and white-listed certain fields. I can see the logs getting printed in the logstash server but logs are not going to elastic search. If i remove the filter it is going to the elastic search. Please advise how to further debug on this issue.

filter {
    kv {
            field_split => "{},?\[\]"
            transform_key => "capitalize"
            transform_value => "capitalize"
            trim_key => "\s"
            trim_value => "\s"
            include_brackets => false   
        }
    prune
    {
        whitelist_names => [ "App_version", "Correlation_id", "Env", "Flow_name", "host", "Instance_id", "log_level","log_thread", "log_timestamp", "message", "patient_id", "status_code", "type", "detail"]
    }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "mule-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

I also need two more suggestion,

I am also trying to use the grok filter in the initial logs and trying to take log level fields(time and log type) from the sample log and send the remaining logs to the KV filter. Is there any reference please share for it. This is what i have tried for it. but getting as _grokparsefailure. I have passed the msgbody to the kv filter with the source option.

grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+%{GREEDYDATA:msgbody}"}
        overwrite => [ "msgbody" ]
    }

I am having message fields inside sample logs as like below. When the data goes to Kibana i can see two message field tag one is with full log and other is with correct message(highlighted). Will the mutate works for this case? Is there any way we can change the full log name as something else ??

[2020-02-10 11:20:07.172] INFO Mule.api [[MuleRuntime].cpuLight.04: [main-api-test].api-main.CPU_LITE @256c5cf5: [main-api-test].main-api-main/processors/0/processors/0.CPU_LITE @378f34b0]: event:00000003 {app_name=main-api-main, app_version=v1, env=Test, timestamp=2020-02-10T11:20:07.172Z, log={correlation_id=00000003, patient_id=12345678, instance_id=hospital, message=Start of System API, flow_name=main-api-main}}

Answer 1

prune filter error

Your prune filter does not have the @timestamp field in the whitelist_names list, your output is date based (%{+YYYY.MM.dd}), logstash needs the @timestamp field in the output to extract the date.

I've ran your pipeline with your sample message and it worked as expected, with the prune filter the message is sent to elasticsearch, but it is stored in an index named mule- without any datetime field.

Without the prune filter your message uses the time when logstash received the event as the @timestamp, since you do not have any date filter to change it.

If you created the index pattern for the index mule-* with a datetime field like @timestamp, you won't see on Kibana any documents on the index that doesn't have the same datetime field.

grok error

Your grok is wrong, you need to escape the square brackets surrounding your timestamp. Kibana has a grok debugger where you can try your patterns.

The following grok works, move your kv to run after the grok and with the msgbody as source.

grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\s+%{LOGLEVEL:loglevel}\s+%{GREEDYDATA:msgbody}"}
    overwrite => [ "msgbody" ]
}
kv {
    source => "msgbody"
    field_split => "{},?\[\]"
    transform_key => "capitalize"
    transform_value => "capitalize"
    trim_key => "\s"
    trim_value => "\s"
    include_brackets => false
}

Just run it with output only to stdout to see the filters you need to change your prune filter.

duplicated message fields

If you put your kv filter after the grok you wouldn't have duplicated message fields since your kv is capitalizing your fields, you will end with a message field containing your full log, and a Message field containing your internal message, logstash fields are case sensitive.

However you can rename any field using the mutate filter.

mutate {
    rename => ["message", "fullLogMessage"]
}