Search code examples
logstashlogstash-json

How to easily promote a JSON member to the main event level?


I'm using an http_poller to hit an API endpoint for some info I want to index with elasticsearch. The result is in JSON and is a list of records, looking like this:

{
  "result": [
     {...},
     {...},
     ...
  ]
}

Each result object in the array is what I really want to turn into an event that gets indexed in ElasticSearch, so I tried using the split filter to turn the object into a series of events instead. It worked reasonably well, but now I have a series of events that look like this:

{ 
  result: { ... }
}

My current filter looks like this:

filter {
  if [type] == "history" {
    split {
      field => "result"
    }
  }
}

Each of those result objects has about 20 fields, most of which I want, so while I know I can transform them by doing something along the lines of

filter {
      if [type] == "history" {
        split {
          field => "result"
        }
        mutate {
           add_field => { "field1" => "%{[result][field1]}"
           #... x15-20 more fields
           remove_field => "result"
        }
      }
    }

But with so many fields I was hoping there's a one-liner to just copy all the fields of the 'result' value up to be the event.


Solution

  • This can be done with a ruby filter like this:

           ruby {
                    code => '
                            if (event.get("result"))
                                    event.get("result").each { |k,v|
                                            event.set(k,v);
                                    }
                                    event.remove("result");
                            end
                    '
            }
    

    I don't know of any way to do this with any of the built in/publicly available filters.