how can i parse only error logs using fluentd

Question

Want to push only error and warning logs to cloudwatch log groups. I want to use fluentd for this approach. This is how my general log file look like.

2022-08-18 06:15:48,983 | 3349 | process_message | INFO | N.A | -1 | -1 | -1 | N.A. | message is empty |

I am using fluent-plugin-cloudwatch-logs plugin. This how my td-agent conf file look like.

```
<source>
@type tail
path /var/log/*/*.log
pos_file /var/log/td-agent/apps.pos
tag disagg-logs
<parse>
@type regexp
  expression /\[\w+\] ERROR\s|(?<message>.*)$/
</parse>
</source>
<match disagg-logs>
@type cloudwatch_logs
log_group_name disagg-logs
log_stream_name disagg-logs
auto_create_stream true
region us-east-1
</match>

With the above configuration file it is pushing even INFO Logs.

Answer 1

able to do with the below regex ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})[\s\|]+(?<pid>\d+)[\s\|]+(?<location>[\w.]+)[\s\|]+(?<level>(INFO|ERROR|WARNING))[\s\|]+(?<uuid>[(\w\d\-|N\.A)]+)[\s\|]+(?<timestart>-?\d+)[\s\|]+(?<timeend>-?\d+)[\s\|]+(?<Id>-?\d+)[\s\|]+(?<type>[(\w\d\-|N\.A)]+)[\s\|]+(?<message>[A-Za-z0-9_ ]+)[\s\|]+$