Search code examples
apache-nifi

Nifi MergeContent truncating/missing flowfile content

The flow file I am working on: ... 1. QueryRecord > 2. SplitJson > 3. ExecuteGraphQuery > 4. MergeContent (defragment)

Step 1 QueryRecord to select a subset of records for processing based on an attribute Step 3 is to get retrieve data based on an attribute of the flow file. Step 4 is set to use Defragment and Binary Concatenation.

In my test case, at step 2, I can verify SplitJson splitted into 3 flow files. At step 3 before MergeContent, the 3 files has fragment.index of 0,1 and 2, all filename, fragment.identifier are matching and fragment.count is 3.

My problem is, the merge content always results in one flowfile with ONLY the content of the flowfile that has index 0 (I have tested 3 times, same result).

I have used MergeContent somewhere else but never ran into this problem.

Solution

  • Although I did not figure out the cause of the issue, the following steps resolved my issue:

    1. Delete the MergeContent processor and drop in another one (Recreating the processor does not resolve the issue)
    2. Work around the issue by converting the json to plain text (part of the intention of the workflow) then do the merging

    I did not test 2. alone as at that point, I already deleted the 'problematic' one.