Google Fluentd Decode Base64

Question

I have log file which have record between two tags RecordStart and RecordEnd the recorded message is base64 encoded I want to decode the message using google-fluentd and so it can send to other services. My Config:

<source>
  @type tail
  path <path_ot>/metrics.log
  pos_file /var/lib/google-fluentd/pos/metrics.pos
  read_from_head true
  format  multiline
  multiline_flush_interval 2s
  format_firstline /^RecordStart/
  format1 /^RecordStart\n(?<record>(\n|.)*)RecordEnd$/
  tag presto_server
</source>

I am not able to figure out how to decode base64 Any help ?

Answer 1

Try using the filter plugin to decode base64 files.

Your config file in this case may look like this:

<source>
  @type tail
  path <path_ot>/metrics.log
  pos_file /var/lib/google-fluentd/pos/metrics.pos
  read_from_head true
  format  multiline
  multiline_flush_interval 2s
  format_firstline /^RecordStart/
  format1 /^RecordStart\n(?<record>(\n|.)*)RecordEnd$/
  tag presto_server
</source>

<filter presto_server>
  type base64_decode
  fields mesg
</filter>

This is an adaptation of the config file I found here.

You may also find this documentation helpful: HYow to modify log records ingested by fluentd.