Behavior of fluentd when it gets parser error. Parser dead letter queue

Recently I have warning message in my fluentd instance when trying to process log message.

The configuration is something like this

<label foo>

  <filter foo.log>
    @type record_transformer

    # Remove all fields other than labels and prettified logs
    renew_record
    keep_keys level, severity, name, actor, environment, deployment, cluster, hostname, ec2_instance_id, az, log_agent, formatted

    enable_ruby
    <record>
      formatted "[ $${record['level']} ] [$${ record.key?('timestamp')? ' timestamp='+record['timestamp'] : '' }$${ record.key?('request_id')? ' request_id='+record['request_id'].to_s : '' }$${ record.key?('actor_id')? ' actor_id='+record['actor_id'].to_s : '' } ] ($${ record.key?('file')? ' file='+record['file'] : '' }$${ record.key?('module')? ' module='+record['module'] : '' }$${ record.key?('function')? ' function='+record['function'] : '' }$${ record.key?('line')? ' line='+record['line'].to_s : '' } )\n$${ record.key?('message')? ''+record['message'] : '' }"
    </record>
  </filter>

  <match ${environment}-backend-v2.log>
    ...
  </match>
</label>

And i got many warning message like this:


2020-10-01 10:37:27 +0000 [warn]: #0 dump an error event: error_class=RuntimeError error="failed to expand `%Q[[ \#{record['level']} ] [\#{ record.key?('timestamp')? ' timestamp='+record['timestamp'] : '' }\#{ record.key?('request_id')? ' request_id='+record['request_id']
 : '' }\#{ record.key?('actor_id')? ' actor_id='+record['actor_id'] : '' } ] (\#{ record.key?('file')? ' file='+record['file'] : '' }\#{ record.key?('module')? ' module='+record['module'] : '' }\#{ record.key?('function')? ' function='+record['function'] :
 '' }\#{ record.key?('line')? ' line='+record['line'].to_s : '' } )\n\#{ record.key?('message')? ''+record['message'] : '' }]` : error = no implicit conversion of Integer into String" location="/opt/bitnami/fluentd/gems/fluentd-1.10.4/lib/fluent/plugin/filter_record_transformer.rb:310:in
 `rescue in expand'" tag="prd-backend-v2.log" time=2020-10-01 10:37:27.050929974 +0000 record=balbasasasdasd

I have 2 questions:

what exactly the behavior of this parser failure if there is no fixing on the parser logic? Does fluentd drop the message?
Can we handle this parser error/exception by process the message regardless, such as sending it to different upstream (dead letter channel)?

Best, Agung

Solution

This event is dropped, unless you've set emit_invalid_record_to_error. Dead letter channel available as an @ERROR label: https://docs.fluentd.org/filter/parser#emit_invalid_record_to_error

Also, there are multiple tricks to make parsing more robust:

For multi-format parsing, like json logs mixed with plaintext logs, use multi-format-parser plugin https://github.com/repeatedly/fluent-plugin-multi-format-parser#for-v10. It will try all formats in a list, and the last format none is a directive to leave a message unparsed, as is. I noticed record=balbasasasdasd in the error trace, you may wanna leave such messages as is.
For record transformers you can use ruby operators to harden this:
```
formatted ${record['main-field'] || record['backup-field'] || record} 
```
and so on.