supporting different kind of logs with Logstash

Question

I'm trying to collect logs using Logstash where I have different kinds of logs in the same file. I want to extract certain fields if they exist in the log, and otherwise do something else.

input{
        file {
                path => ["/home/ubuntu/XXX/XXX/results/**/log_file.txt"]
                start_position => "beginning"
         }
}
filter {
        grok{
                         match => { "message" => ["%{WORD:logger} %{SPACE}\-%{SPACE} %{LOGLEVEL:level} %{SPACE}\-%{SPACE} %{DATA:message} %{NUMBER:score:float}",
                                "%{WORD:logger} %{SPACE}\-%{SPACE} %{LOGLEVEL:level} %{SPACE}\-%{SPACE} %{DATA:message}"]}
        }
}

output{
         elasticsearch {
                 hosts => ["X.X.X.X:9200"]
        }
        stdout { codec => rubydebug }
}

for example, log type 1 is: root - INFO - Best score yet: 35.732

and type 2 is: root - INFO - Starting an experiment

One of the problems I face is that when a message doesn't contain a number, the field still exists as null in the JSON created which prevents me from using desired functionalities in Kibana.

Answer 1

One option, is just to add a tag on logstash when field is not defined to be able to have a way to filter easily on kibana side. The null value is only set during the insertion in elasticsearch (on logstash side, the field is not defined)

This solution looks like this in you case :

    filter {
            grok{
                             match => { "message" => ["%{WORD:logger} %{SPACE}\-%{SPACE} %{LOGLEVEL:level} %{SPACE}\-%{SPACE} %{DATA:message} %{NUMBER:score:float}",
                                    "%{WORD:logger} %{SPACE}\-%{SPACE} %{LOGLEVEL:level} %{SPACE}\-%{SPACE} %{DATA:message}"]}
            }
            if ![score] {
                             mutate { add_tag => [ "score_not_set" ] }
            }

    }