Search code examples
fluentd

Fluentd does not parse nested json array


I have the following json message on Fluentd input

{"foo":{"bar":{"abc":"[\n {\n \"ip\":\"192.168.1.1\",\n \"hostname\":\"pc\",\n \"mac\":\"01:02:03:04:05:06\"\n} \n]"}}}

And want to get the output message

{"foo":{"bar":{"abc":[{"ip":"192.168.1.1", "hostname":"pc", "mac":"01:02:03:04:05:06}]"}}}

I'm trying to parse it with the filter

<filter **>
  @type parser
  key_name foo
  reserve_data true
  remove_key_name_field false
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>  

But without any effect. The output is the same as the input.

When I try

<filter **>
  @type parser
  key_name $['foo']['bar']['abc']
  <parse>
    @type none
  </parse>
</filter>

<filter **>
  @type parser
  key_name message
  <parse>
    @type json
  </parse>
</filter>

it "works" but of course I get only

[{"ip":"192.168.1.1","hostname":"ps","mac":"01:02:03:04:05:06}]

Thank you for your suggestions.


Solution

  • this parses unparsed nested json

    <filter **>
      @type parser
      key_name $['foo']['bar']['abc']
      hash_value_field parsed_abc
      reserve_data true
      remove_key_name_field true
      <parse>
        @type json
      </parse>
    </filter>
    
    <filter **>
      @type record_transformer
      enable_ruby true
      <record>
        parsed_abc ${record['foo']['bar']['abc'] = record['parsed_abc']}
      </record>
      remove_keys parsed_abc
    </filter>