Search code examples
elasticsearchlogstashrethinkdb

Logstash mutate add all fields from json

I am using a Logstash plugin (logstash-input-rethinkdb). This plugin grabs all the edits in the database and outputs an json object containing the following structure: 

{
      "db":"itjobs",
      "table":"countries", 
      "old_val":null,
      "new_val":{
           "code":"USA3",
           "country":"USA3",
           "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0"
      },
      "@version":"1",
      "@timestamp":"2016-12-19T19:54:08.263Z"
 }

I insert this doc in elasticsearch.
But the problem is that in elastic i get the same structure with -> new_val:{code:''}

I need to do a filter or something that will extract everything from new_val and add it to the root

I tried the filter json but this get a string and the plugin already outputs json

After edit i need it to look like:

{
      "db":"itjobs",
      "table":"countries", 
      "code":"USA3",
      "country":"USA3",
      "id":"7c8c9e4e-aa37-48f1-82a5d624cde4a3a0"
      "@version":"1",
      "@timestamp":"2016-12-19T19:54:08.263Z"
 }
Solution

  • You can use the ruby filter:

    filter {
    ruby {
        code => "
            event['new_val'].each {|k, v|
                event[k] = v
            }
            event.remove('new_val')
        "
    }
}

    Tested with Logstash 2.2, it should be different for version 5+.