I am using Filbeat for log aggregation, which takes the logs to Kibana. Below is my error message that needs to be directed to Kibana:
2017-04-17 15:45:47,154 [JCO.ServerThread-8] ERROR com.webservice.AxisWebServiceClient - Client error
2017-04-17 15:45:47,154 [JCO.ServerThread-8] ERROR com.webservice.AxisWebServiceClient - The XML request is invalid. Fix the request and resend.
310,273,990
310,292,500
360,616,489
2017-04-04 12:47:09,362 [JCO.ServerThread-3] INFO com.app.Listener - End RFC_CALCULATE_TAXES_DOC
2017-04-04 12:47:09,362 [JCO.ServerThread-3] DEBUG com.Time - RFC_CALCULATE_TAXES_DOC,DEC[2],Total Time,39
i want only to have 2017-04-17 15:45:47,154 [JCO.ServerThread-8]ERROR and lines below the error to be send to Kibana, but i do get the INFO part as well
Below is filbeat.yml file
filebeat:
prospectors:
-
paths:
- /apps/global/vertex/SIC_HOME_XEC/logs/sic.log
input_type: log
exclude_lines: ['^INFO']
#include_lines: 'ERROR'
multiline:
pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s\[[A-Za-z0-9.-]*\]\s[E]RROR'
negate: true
match: after
Request veterans help to select only the ERROR message pattern using regex.
In order to extract the error messages as a group, you'll need to modify your regex as following:
^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3}\s\[[A-Za-z0-9.-]*\]\sERROR (\w.+)
Explanation:
(\w.+)
This creates a group with all characters and the dot character, which captures the error message.