Search code examples
logstash-grokgrok

Netfilter syslog message Grok pattern

I've got a Netfilter syslog log message and am attempting to figure out why a grok pattern is failing to parse a message successfully. I want to be able to format it via. a script in order for it to parse successfully. Here's a sample of the syslog message:

<5>Apr 24 12:07:59 CoreNetwork01 kernel: NF:DROP:IN=wan2 OUT=lan2 MAC=00:90:0b:93:24:80:7c:21:0e:0d:38:84:08:00 SRC=114.123.102.228 DST=201.112.128.68 LEN=34 TOS=0x00 PREC=0x00 TTL=23 ID=52031 PROTO=TCP TYPE=8 CODE=0 ID=51033 SEQ=0 MARK=0x130000

The grok pattern that I am attempting to format it to conform too is as follows:

<%{INT}>( )?%{SYSLOGTIMESTAMP:event_time} %{HOST:src_host} %{DATA:product_event_type}: \\[.*?\\] %{NOTSPACE}( %{WORD:action})? %{GREEDYDATA:kv_data}

I've tested with the tool here https://www.javainuse.com/grok and figured out that the pattern will work correctly up until this section:

`\\[.*?\\] %{NOTSPACE}( %{WORD:action})? %{GREEDYDATA:kv_data}

What should the original syslog message look like in order for the syslog message to parse correctly?

Solution

  • It would seem [\\[.*?\\] regex group block is breaking the Grok pattern. If it is removed the pattern looks like and will work:

    <%{INT}>( )?%{SYSLOGTIMESTAMP:event_time} %{HOST:src_host} %{DATA:product_event_type}: %{NOTSPACE}( %{WORD:action})? %{GREEDYDATA:kv_data}

    Unfortunately; this pattern is baked into the Chronicle IPTable parser and I cannot tweak or change it but after acquiring the parser via the chronicle_cli there is luckily another match condition within the grok pattern:

    grok {
    match => {
      "message" => [
        "<%{INT}>( )?%{SYSLOGTIMESTAMP:event_time} %{HOST:src_host} %{DATA:product_event_type}: \\[.*?\\] %{NOTSPACE}( %{WORD:action})? %{GREEDYDATA:kv_data}"
        "%{SYSLOGTIMESTAMP:event_time} %{HOST:src_host} %{WORD:product_event_type}:.*%{WORD:action}: %{GREEDYDATA:kv_data}"
      ]
    }
    overwrite => ["kv_data", "event_time", "src_host", "action"]
  }

    If I format the message (Remove header, add in additional event type verb) slightly from it's original format it looks like this:

    Apr 24 12:07:59 CoreNetwork01 kernel: kernel: NF:DROP:IN=wan2 OUT=lan2 MAC=00:90:0b:93:24:80:7c:21:0e:0d:38:84:08:00 SRC=114.123.102.228 DST=201.112.128.68 LEN=34 TOS=0x00 PREC=0x00 TTL=23 ID=52031 PROTO=TCP TYPE=8 CODE=0 ID=51033 SEQ=0 MARK=0x130000

    It will now parse against this grok pattern!

    %{SYSLOGTIMESTAMP:event_time} %{HOST:src_host} %{WORD:product_event_type}:.*%{WORD:action}: %{GREEDYDATA:kv_data}

    It works!

    {
  "product_event_type": "kernel",
  "MONTH": "Apr",
  "HOUR": "12",
  "action": "kernel",
  "TIME": "12:07:59",
  "MINUTE": "07",
  "SECOND": "59",
  "src_host": "CoreNetwork01",
  "event_time": "Apr 24 12:07:59",
  "MONTHDAY": "24",
  "kv_data": "NF:DROP:IN=wan2 OUT=lan2 MAC=00:90:0b:93:24:80:7c:21:0e:0d:38:84:08:00 SRC=114.123.102.228 DST=201.112.128.68 LEN=34 TOS=0x00 PREC=0x00 TTL=23 ID=52031 PROTO=TCP TYPE=8 CODE=0 ID=51033 SEQ=0 MARK=0x130000"
}