Fluentd regex filter removes other keys

Question

I'm getting a message into fluentd with a few keys already populated from previous stages (fluent-bit on another host). I'm trying to parse the content of the log field as follows:

# Parse app_logs
<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>

It works (kind of), as it extracts the fields as expected. That said, it removes all the other fields that were there before.

Example message before the filter:

filter.app.backend.app_logs: {"docker.container_name":"intranet-worker","docker.container_id":"98b7784f27f93a056c05b4c5066c06cb5e23d7eeb436a6e4a66cdf8ff045d29f","time":"2022-06-10T17:00:00.248932151Z","log":"org-worker  2022-06-10 19:00:00,248 INFO     briefings.py:check_expired_registrations Checking for expired registrations\n","docker.container_image":"registry.my-org.de/org-it-infrastructure/org-fastapi-backend/backend-worker:v0-7-11","stream":"stdout","docker.container_started":"2022-06-10T14:57:27.925959889Z"}

After the filter, the message looks like this (its a slightly different one, but same stream):

filter.app.backend.app_logs: {"module":"mksp-api","severity":"DEBUG","file":"authToken.py","function":"verify_token","message":"Token is valid, checking permission"}

So only the parsed fields are kept, the rest is removed. Can I somehow use that filter to add the fields to the message, instead of replacing it?

Answer 1

Actually, this scenario is described in the documentation, its not part of the regexp documentation but of the corresponding parser filter documentation:

reserve_data Keeps the original key-value pair in the parsed result.

Therefore, the following configuration works:

<filter filter.app.backend.app_logs>
  @type parser
  key_name log
  reserve_data true
  <parse>
    @type regexp
    expression /^(?<module>[^ ]*) *(?<time>[\d ,-:]*) (?<severity>[^ ]*) *(?<file>[\w\.]*):(?<function>[\w_]*) (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S,%L
  </parse>
</filter>