Search code examples
apache-nifi

Split values inside attribute into multiple flowfiles

I have a flowfile with empty content and some attributes.

I'm interested in the attribute ids. It contains list of ids with , separator.

1018866556,1018878837,1018522766,1018522773,1018522788,1018522790,1018522797,
1018522959,1018522963,1018522968,1018522972,1018522981,1018511143,1018511174

I'm using this attribute in InvokeHTTP processor, but sometimes errors occur due to a large amount of data.

If an error occurs I want to split values inside this attribute. I should get two flowfiles instead of one with the same (or close to it) number of ids inside each flowfile.

So, after transformation I expect:

  1. Flowfile #1 with ids 1018866556,1018878837,1018522766,1018522773,1018522788,1018522790,1018522797
  2. Flowfile #2 with ids 1018522959,1018522963,1018522968,1018522972,1018522981,1018511143,1018511174

If error still occurs, then split them again, etc... Like binary search but without sorting.

How to reach it?

Solution

  • script for ExecuteGroovyScript processor:

    def ff=session.get()
if(!ff)return

def idList = ff.ids?.split(',')
if ( idList.size()>1 ){
    def sublistSize = (idList.size()/2).round() as int
    def ffList = idList.collate(sublistSize).collect{sublist-> 
            ffOut = ff.clone(true)
            ffOut.ids = sublist.join(',')
            ffOut
        }
    REL_SUCCESS << ffList
    ff.remove()
}else{
    REL_SUCCESS << ff
}