Parsing a JSON Array into separate JSON events Losgstash?

Question

I currently have a situation in which logstash pulls a JSON array from azure event hubs which i need to split into multiple events.

The logs i get from the event hub look like this:

{"records": [{JSON LOG},{JSON LOG},{JSON LOG},...,{JSON LOG}]}

I have tried using split and the json filter but i can't quite seem to get it to work. I basically want to split up the array so that logstash sends to elastic each of the JSON Logs in records as a separate event, parsed as json.

I also need to rename/parse the individual JSON logs into ECS so currently think i need to parse records as json and then parse the output as json before doing some mutate rename filters before sending to elastic, unless it would be easier to just do the parsing as JSON in logstash with an elastic index pipeline for the parsing to ECS.

My current filter section is:

filter {    
    #Split results into individual events
    json {
        source => "message"
    }

    #add a target_index field for the final index to send to
    mutate {
        replace => [ "[@metadata][target_index]", "logs-eventhub" ]
    } 
}

Would anyone be able to provide some insight into how to do multiple json parses in logstash so that i get each record in it's own event parsed as JSON.

Answer 1

If your input parses the data as JSON, you don't need a json filter, you'd simply need to use the split filter like this:

filter {    
    #Split results into individual events
    split {
        source => "records"
    }

    #add a target_index field for the final index to send to
    mutate {
        replace => [ "[@metadata][target_index]", "logs-eventhub" ]
    } 
}

What the split filter will do is to clone your original event, split the records array and place each sub-record into a clone of the original event.

Try it out!!