Modify fluentd json with record_transformer plugin

Question

I have logs like this:

{
    "posts": {
        "key1": "value123",
        "key2": "abcdge123",
        "key3": "abcdge345",
        "....": "....",
        "something": "something"
    },
    "execute_time": 123,
    "code": 200,
}

Here's fluentd configuration for record_transformer:

<filter tag.hellow>
  @type record_transformer
  enable_ruby true
  <record>
    posts ${ if record['posts'].has_key?('key1'); Base64.strict_encode64(record['posts']['key1'].to_s); end }
  </record>
</filter>

This deletes the posts field and recreates so that all the other keys in the posts field are lost i.e.:

{
    "posts": {
        "key1": "base64XXXXXXX"
    },
    "execute_time": 123,
    "code": 200,
}

This is the desired log output:

{
    "posts": {
        "key1": "base64XXXXXXX",
        "key2": "abcdge123",
        "key3": "abcdge345",
        "....": "....",
        "something": "something"
    },
    "execute_time": 123,
    "code": 200,
}

Is there any way for this use-case?

Thanks!

Answer 1

You need to return the complete posts object after modification like this:

posts ${ if record['posts'].has_key?('key1'); record['posts']['key1'] = Base64.strict_encode64(record['posts']['key1'].to_s); record['posts']; end }

In its current form i.e.:

posts ${ if record['posts'].has_key?('key1'); Base64.strict_encode64(record['posts']['key1'].to_s); end }

it returns the first object only after modification which is assigned to posts and that is what you're seeing.