I'm a newbie to fluentd and trying to parse logs with my limited knowledge. we have diverse applications which write logs in different formats. I have written a source to parse the log. since I don't have control on to maintain a standard on the logs. I won't be able to change them.
However i'm not sure how to handle when the pattern doesn't match. Basically if the pattern doesn't match I want to send the log as is.
I tried looking for documentation and couldn't find much Thanks in advance. here is my source
<source>
@type tail
path /Users/amjad/testfluent/*.log
read_from_head true
format multiline
format_firstline /\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}/
format1 /(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}) +(?<log_level>[aA-zZ]*) \[(?<service_name>.+),(?<trace_id>.*),(?<span_id>.*),(?<zipkin>[aA-zZ]*)\] (?<message>(.|\s)*)/
tag kubernetes.apps*
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>
The source directive will only tail the log file and try to generate log events that need to be parsed and handled afterward. since you are using a multiline format, the source will concatenate the lines that match the format into a single event, otherwise, it will generate a log event for each line. later then you can match all these events and handle them like this
<match kubernetes.apps*>
@type xx
<macth>
<match **.**>
@type xx
<macth>