Apache NiFi: PutElasticSearchHttp is not working, with blank error

Question

I currently have Elasticsearch version 6.2.2 and Apache Nifi version 1.5.0 running on the same machine. I'm trying to follow the Nifi example located: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html except instead of storing to Hive, I want to store to Elasticsearch.

Initially I tried using the PutElasticsearch5 processor but I was getting the following error on Elasticsearch:

Received message from unsupported version: [5.0.0] minimal compatible version is: [5.6.0]

When I tried Googling this error message, it seemed like the consensus was to use the PutElasticsearchHttp processor. My Nifi looks like:

And the configuration for the PutElasticsearchHttp processor:

When the flowfile gets to the PutElasticsearchHttp processor, the following error shows up:

PutElasticSearchHttp failed to insert StandardFlowFileRecord into Elasticsearch due to , transferring to failure. 

It seems like the reason is blank/null. There also wasn't anything in the Elasticsearch log.

After the ConvertAvroToJson, the data is a JSON array with all of the entries on a single line. Here's a sample value:

{"City": "Athens", 
 "Edition": 1896, 
 "Sport": "Aquatics", 
 "sub_sport": "Swimming",
 "Athlete": "HAJOS, Alfred", 
 "country": "HUN", 
 "Gender": "Men", 
 "Event": "100m freestyle", 
 "Event_gender": "M", 
 "Medal": "Gold"}

Any ideas on how to debug/solve this problem? Do I need to create anything in Elasticsearch first? Is my configuration correct?

Answer 1

I was able to figure it out. After the ConvertAvroToJSON, the flow file was a single line that contained a JSON Array of JSON indices. Since I wanted to store the individual indices I needed a SplitJSON processor. Now my Nifi looks like this:

The configuration of the SplitJson looks like this: