Search code examples
elasticsearchlogstashelastic-stackfilebeat

Filebeat multiline filter is not working?

I am trying to read files from filebeat and push them into logstash. Before pushing them, I am trying to merge events which contain java stack trace. I tried this filter but its not working out.

filebeat.prospectors:
- type: log
  paths:
- /mnt/logs/myapp/*.log

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

This is the example of logs which I am trying to push to logstash. I want to merge stack trace events to event with timestamp which came before it

2019-02-18 17:08:44 augmentos  http-nio-8090-exec-4 INFO c.a.s.c.a.a.CommonCorpAuthAspect@validateAuth:68 - Header Name: connection And Header Value : keep-alive
2019-02-18 17:08:44 augmentos  http-nio-8090-exec-4 INFO c.a.s.c.a.a.CommonCorpAuthAspect@validateAuth:118 - Permission status is true
2019-02-18 17:08:44 augmentos  http-nio-8090-exec-4 INFO c.a.s.c.a.c.AssetADVActionResource@emailAssets:92 - User testqa is sending Asset on Email.
2019-02-18 17:08:47 augmentos  http-nio-8090-exec-4 INFO c.a.spectra.aws.utils.S3ServiceUtil@generateSignedUrl:48 - Generating pre-signed URL with timeout 604800000
2019-02-18 17:08:56 augmentos  http-nio-8090-exec-4 ERROR c.a.s.c.a.a.CommonCorpAuthAspect@validateAuth:131 - Authorization Aspect error
java.lang.NullPointerException: null
        at com.ad2pro.spectra.core.acs.service.EmailHandlerService.getAssetFiles(EmailHandlerService.java:140)
        at com.ad2pro.spectra.core.acs.service.EmailHandlerService.emailAsset(EmailHandlerService.java:63)
        at com.ad2pro.spectra.core.acs.controllers.Test.emailAssets(AssetADVActionResource.java:104)
        at com.ad2pro.spectra.core.acs.controllers.Test$$FastClassBySpringCGLIB$$bfc0b3c1.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
2019-02-18 17:08:44 augmentos  http-nio-8090-exec-4 INFO c.a.s.c.a.a.CommonCorpAuthAspect@validateAuth:68 - Header Name: connection And Header Value : keep-alive
2019-02-18 17:08:44 augmentos  http-nio-8090-exec-4 INFO c.a.s.c.a.a.CommonCorpAuthAspect@validateAuth:118 - Permission status is true

After merging events, I suppose message attribute should have both events messages concatenated if I am right, please help on this.

Solution

    1. Your regular expression should be good. See https://www.elastic.co/guide/en/beats/filebeat/7.5/_test_your_regexp_pattern_for_multiline.html how you can actually test this.

    2. It might just be a copy paste error, but the indentation of your YAML file doesn't look right. Also check your Filebeat logs for errors. This should be:

       - type: log
   paths:
     - /mnt/logs/*.log
   multiline:
     pattern: '^\['
     negate: true
     match: after