Search code examples
elasticsearchlogstashlogstash-groklogstash-configuration

Is it possible to index only the matched log lines of grok in Logstash?

I'm having a log file, which actually has INFOs' and ERRORs. So I tried to match only the needful INFOs by using the grok filter. So this is how my log lines look like. Few of them from the file.

And this is how my grok look like in my logstash conf:

grok {
        patterns_dir => ["D:/elk_stack_for_ideabiz/elk_from_chamith/ELK_stack/logstash-5.1.1/bin/patterns"]
        match => { 
            "message" => [
                "^TID\: \[0\] \[AM\] \[%{LOGTIMESTAMPTWO:logtimestamp}]%{REQUIREDDATAFORAPP:app_message}",
                "^TID\: \[0\] \[AM\] \[%{LOGTIMESTAMPTWO:logtimestamp}]%{REQUIREDDATAFORRESPONSESTATUS:response_message}"
            ]   
        }
    }

The pattern seems to be working fine. I could provide the pattern if required.

I've got two questions. One is I wanted only the grok matched lines to be sent to the index, and prevent Logstash from indexing the non-matched ones and the second is to prevent Logstash from showing the message in every single ES record.

I tried using the overwrite as such under the match but still no luck:

overwrite => [ "message" ]

All in all what I need to see in my indice are the messages (app_message, response_message from the above match), which should match the above two conditions. Where as now, all the lines are getting indexed.

Is it possible do something like this? Or does Logstash index all of them by default?

Where am I going wrong? Any help could be appreciated.

Solution

  • Indexing only the lines you want is pretty straightforward.

    By default, if grok fails to match anything it will add the tag _grokparsefailure.

    Now what you want to do is check for that tag and then use the drop filter.

    if "_grokparsefailure" in [tags] {
  drop { }
}

    sysadmin1138 mentioned that you can select between different grok filters by adding tag_on_failure => ['_INFOFailure' ]

    For your sencond question you can simply remove the field. Every filter has a remove_field option but I use the mutate filter to indicate that I am manipulating the message.

    mutate {
  remove_field => ["message"]
}