Grok pattern for tomcat logs gives compile error on Grok Debugger

Question

I am trying to setup ELK for a Java application. The tomcat logs are produced using log4j. To write a test a pattern, I am using Grok Debugger. But on the debugger it always shows

Compile ERROR

My log sample:

YYYY-MM-DD HH:MM:SS,SSS INFO : [so-me-uni-que-id] com.xx.xx.xx.xx.xx - log message here

My grok filter:

filter {   if [type] == "tomcat" {     grok {       match => { "message" => "%{TOMCATLOG}" }     }     date {       match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]     } } }

My pattern:

TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{UNIQUEID:uniqueid}\| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

Answer 1

The basic issue is that your pattern doesn't match your input. Look at the beginning:

YYYY-MM-DD HH:MM:SS,SSS INFO : [so-me-uni-que-id]

%{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{UNIQUEID:uniqueid}\|

Your pattern has escaped pipes ("|"), but the input doesn't use them.

I also don't see that TOMCAT_DATESTAMP is in the default patterns, but maybe it's buried somewhere.

Start at the left side, matching one piece at a time in the debugger.

%{TIMESTAMP_ISO8601} %{WORD:level} : \[%{GREEDYDATA:uniqueid}\]

Then keep working your way across, grabbing more stuff into your pattern. Note that literals (":" and the escaped "[") become part of your pattern.

Good luck!