Search code examples
logstashelastic-stacklogstash-grok

Skip first few lines of log using grok filter plugin

Is it possible to remove the whole first line starting with date-time in the following log? I'm only interested in the second line.

Dec 09, 2019 12:55:22 PM hudson.slaves.CommandLauncher launch
INFO: agent launched for ci-vd6

How can I do the above using the grok filter plugin ?

Solution

  • You can skip lines like that:

    if [message] =~ /2019/ {
  drop { }
}

    Now you need to change regex to drop lines when you need it only.

    EDIT: For example baudsp suggests the regex could look like that:

    \b(?:[Jj]an(?:uary|uar)?|[Ff]eb(?:ruary|ruar)?|[Mm](?:a|ä)?r(?:ch|z)?|[Aa]pr(?:il)?|[Mm]a(?:y|i)?|[Jj]un(?:e|i)?|[Jj]ul(?:y)?|[Aa]ug(?:ust)?|[Ss]ep(?:tember)?|[Oo](?:c|k)?t(?:ober)?|[Nn]ov(?:ember)?|[Dd]e(?:c|z)(?:ember)?)\b \d{2}, \d{4}

    I mean scenario when you receive message line by line, so you receive:

    Dec 09, 2019 12:55:22 PM hudson.slaves.CommandLauncher launch

    You drop it.

    You receive line:

    INFO: agent launched for ci-vd6

    you keep it.

    Edit: to split lines:

    Split the message regarding for example newline if possible. Enable special signs in logstash.yml: config.support_escapes: true and add filter 

    filter {
  split {
  field => "message"
  terminator => "\n" 
 }
}

    And now process them one by one.