I've got an issue with flowfiles passing through a mergerecord.
Here is the flow (click on link for image):
I've tried most of the permutations of the configuration settings but can't seem to get flowfiles out of the queue no matter what I do:
Does anyone know what could be blocking this mergerecord from passing flowfiles? It seems the flowfiles are currently "text" files, would they need to be JSON for the mergerecord to group correctly?
The Merge is correlating on TableName - meanining it is only going to merge flowfiles where the TableName attribute is the same value.
However, you only have 10 total bins - meaning if 10 flowfiles come in with table1,2,3,4,5,6,7,8,9,10 you have maxed out your bins, so any FlowFiles with table11,12,13,14, etc. aren't going to get merged until a bin frees up. They will just sit in the queue and wait.
Further, your Merge config is also only set with Min 1 and Max 1000 - meaning you need 1000 records with TableName = table1, before those files are merged and the bin is released.
With 5000 FlowFiles making up 3MB, I'm going to assume there aren't many Records per FlowFile, so you aren't filling up 1000 Records and releasing any bins.
So, double check that your TableName attribute is being set as you expected, and consider modifying the setting for controlling the merge. You could lower the Max Records from 1000 to trigger sooner, you could add a Max Size, or you could add a Max Age to time-bound it.
