Search code examples
regexlogstashlogstash-grok

Logstash Optional Fields


I have log examples like this where uptime isn't always a field:

Oct 17 08:41:33 host-name 2y40w: %SEC-6-IPACCESSLOGP: list 105 permitted tcp 1.2.3.4(17774) -> 0.0.0.0(22), 1 packet
Oct 17 13:32:26 host-name %SEC-6-IPACCESSLOGP: list 105 permitted tcp 1.2.3.4(56200) -> 0.0.0.0(22), 1 packet

If i do regex like this:

   %{SYSLOGTIMESTAMP:log_server_timestamp} %{IPORHOST:cisco_host (%{DATA:uptime}:)? %{NOTSPACE:msgid}: \list %{NUMBER:acl} %{WORD:action} %{WORD:protocol} %{NOTSPACE:source}+\(%{DATA:src_port}\) \-> %{NOTSPACE:dest}+\(%{DATA:dest_port}\), %{NUMBER:packets}

With a space between:

(%{DATA:uptime}:)? %{NOTSPACE:msgid}:

It will parse the log that includes uptime successfully, but not the log without uptime.

Without a space like:

(%{DATA:uptime}:)?%{NOTSPACE:msgid}:

It will parse the message that does not include the uptime.

I am a bit rusty with my regex so was hoping someone could help me figure out why this isn't taking as I really don't like the idea of adding another level of nested if statements when a conditional field like this should be available in regex.

edit:

also tried:

(?<uptime>[ywd0-9]+):

Which does not work either


Solution

  • You may enclose the whole part of the uptime pattern + whitespace with an optional group:

    %{SYSLOGTIMESTAMP:log_server_timestamp} %{IPORHOST:cisco_host}( %{DATA:uptime}:)? %{NOTSPACE:msgid}: \list %{NUMBER:acl} %{WORD:action} %{WORD:protocol} %{NOTSPACE:source}+\(%{DATA:src_port}\) \-> %{NOTSPACE:dest}+\(%{DATA:dest_port}\), %{NUMBER:packets}
                                                                  ^^               ^^