Search code examples
elasticsearchlogstashkibanalogstash-grok

Continue with Groking On Failure

I have a grok expression slice my log4j file to make it available for kibana via elastic search. I'm starting with a simple grok expression as I'm still learning

match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+%{IP:ip}"}

Here in this case, some of my log files doesn't have have an IP filed. So the grok debugger shows a "no match".

Does that mean that I'm going to miss that specific line being parsed is going to drop, because of these match?

How can I continue with the matching process even when there is a some mismatch?

Sample log messages created by logstash

 Eg1. Without IP   {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetTripsByFlightNumber</con:node><con:pipeline>GetTripsByFlightNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.axsbagtracing.flight.ws.qantas.FlightScheduleRequestBindingStub.getTripsByFlightNumber(FlightScheduleRequestBindingStub.java:1563)\n\tat {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:00:02,319 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetTripsByFlightNumber</con:node><con:pipeline>GetTripsByFlightNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.axsbagtracing.flight.ws.qantas.FlightScheduleRequestBindingStub.getTripsByFlightNumber(FlightScheduleRequestBindingStub.java:1563)\n\tat com.acme.bagassist.common.scheduler.InboundListFlightDataProcessor.callOGSForFlightTime(DataProcessor.java:398)\n\tat com.acme.bagassist.common.scheduler.InboundListFlightDataProcessor.processInboundFlightData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied","_grokparsefailure"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7568,"source":"C:\\logs\\applog_x16.log","type":"log","host":"LVRJ8YRJX1"}


 Eg2. With IP      {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:07:36,535 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:504 ) WTSDK- Command: V.1\nVHDG.WA/I5BAGXS/E�/PQF7436\nVGZ.\nVQF////33080\nWM DAH PERQF11417.FAPAX/BAG/RTI/CLM/OSI","@version":"1","tags":["multiline","beats_input_codec_multiline_applied","_grokparsefailure"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7834,"source":"C:\\logs\\applog_x16.log","type":"log","host":"LVRJ8YRJX1"}

Please find an except from my log file below:

2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > 
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server
 faultSubcode: 
 faultString: 0005: No Data matched the criteria Specified
 faultActor: 
 faultNode: 
 faultDetail: 
    {http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>getNumber</con:node><con:pipeline>getNumber_response</con:pipeline><con:stage>Create Number Response</con:stage><con:path>response-pipeline</con:path></con:location>

0005: No Data matched the criteria Specified1
    at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
    at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
2013-04-05 00:07:36,535 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:504 ) WTSDK- Command: V.1
ACDG.WA/ACMEXS/E…/PQF7436
VQZ.
VMF////33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36,557 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:505 ) WTSDK- PID: PQF7436
2013-04-05 00:07:40,120 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:517 ) WTSDK: Response Time before parsing using PID PQF7436 == 3560 ms
2013-04-05 00:07:40,126 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:547 ) WTSDK: Response string after parsing: WM DAH PERQF11417

Thanks, San

Solution

  • The grokparsefailure means that none of the patterns that you provided to grok could be successfully applied to the message that was received. The rest of your filters and outputs will run, but any fields that you were expecting to be created by the grok{} won't happen.

    In regular expressions, you can make pieces of them optional, e.g.

    (?:%{URIPATHPARAM})?

    EDIT:

    Aside from the optional IP field, you also have fairly random whitespace in your data. %{SPACE} will match any whitespace, and is typically more readable than "\s*".

    A few minutes in the debugger led to this:

    %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}%{SPACE}\[%{NOTSPACE:program}%{SPACE}]%{SPACE}(?:%{IP:ip})?%{SPACE}\(%{SPACE}%{NOTSPACE:coderef} \)

    which works for both inputs.