fluentd record_transformer - wrapping $[record] in additional json objects

Question

I'm changing how I'm consuming GCP logs from receiving a PubSub subscription push directly into my log analytics tool to pulling the PubSub subscription with Fluentd and then pushing the logs into the log analytics tool. The problem is that now I'm not getting some additional json wrapper objects around the log data and I need to put them back or the change will break all our dashboards and scheduled searches.

I'm trying to use record_transformer to add the wrapper objects and it's not clear how to do this.

Given a message like this:

{"foo": "bar"}

I need it to come out like:

{
  "message": {
     "data": {
        "foo": "bar"
      }
   }
}

(I don't need it pretty formatted, I just did that for readability here.)

I tried this:

<filter gcp.logs>
  @type record_transformer
  <record>
    message data $[record]
  </record>
</filter>

But that doesn't do the nesting of json objects that I was hoping for.

Any pointers in the right direction would be much appreciated.

Answer 1

You can format the record and wrap it with the needed keys with the following configurations

<filter gcp.logs>
  @type record_transformer
  enable_ruby true
  renew_record true
  <record>
    message ${ {data: record } }
  </record>
</filter>