Search code examples
logstashelastic-stacklogstash-grok

logstash filter , unfiltered lines

I am new to Logstash filter and going through different blogs and links to understand in detail. I have few questions which are still unanswered.

. If my log file has different log pattern e.g.

2017-01-30 14:30:58 INFO ThreadName:33 - {"t":1485786658088,"h":"abcd1234", "l":"INFO", "cN":"org.logstash.demo", "mN":"getNextvalue", "m":"fetching next value"} 2017-01-30 14:30:58 INFO AnotherThread:33 -my log pattern is different

I have below filter which is successfully filtering line 1 of the log

 grok
 {  
      match => [ "message", "%{TIMESTAMP_ISO8601:LogDate} %{LOGLEVEL:loglevel} %{WORD:threadName}:%{NUMBER:ThreadID} - %{GREEDYDATA:Line}" ] 
  }
  json
  {
      source => "Line" 
  }
  1. what will happen with the lines which can not be filtered using filter pattern?
  2. Is there any way to capture all the lines which were not filtered and send to elasticSearch ?
  3. Is there any good reading material where I can read about Input, Filter, Output plugins with the examples ?
Solution

  • To answer your questions:

    1. The lines which cannot be filtered using grok would end up in a grok_parsefailure. Make sure you handle it by dropping the lines which don't actually match the filter criteria.

    2. As far as I know you can't capture them separately and push it to ES. Maybe for this, you can have multiple grok patterns so that you can filter it out and send it to different ES indices thereafter.

    3. I've added the links in the comment above.

    This SO could come in handy. Hope it helps!