Search code examples
fluentd

How to expand JSON in Fluentd?


Is there a way to filter out the nested JSON string out into separate fields in fluentd?

Current JSON:

 { 
   Value1: "something",
   Value2: "something",
   Message:{
             Value3: "Something",
             Value3: "Something"
          }
 }

What I want (or something similar):

 { 
   Value1: "something",
   Value2: "something",
   Message.Value3: "Something",
   Message.Value3: "Something"
          
 }

The JSON doesn't have to be flatten like in above example but I do want the values to be in their own separate fields(columns) when they reach elasticsearch. In other wards, I want to split the single long Message string into multiple fields contained within it.


Solution

  • Try this:

    <source>
    ...
    </source>
    
    <filter myapp**>
      @type parser
      key_name Message
    
      format multi_format
      <pattern>
        format json # try parsing json in the first place
      </pattern>
      <pattern>
        format none # leave as is if this is not json (plaintext)
      </pattern>
    
      reserve_data true # keep the original Message field in case anything go wrong
    </filter>
    
    <match myapp**>
    ...
    </match>
    

    Multi-format parser: https://github.com/repeatedly/fluent-plugin-multi-format-parser