Search code examples
apache-nifi

Wait for all the files arrival-NiFi

Is there any processor in Nifi that waits for the all the files to arrive and then put those files into HDFS.

For example: If there are total 5 files to be fetched using SFTP but we received only 3 files, I want NiFi to wait till 5 files arrived and then put those 5 files into HDFS using PUTHDFS.

Thank you for your anwsers

Solution

  • You can use List* processors with a Record Writer and use a MergeRecord processor to wait for a specific number of files.

    1. Use a ListSFTP processor. Set the Record Writer attribute. You can use anyone.
    2. Connect the success to a MergeRecord processor with maximum and minimum bin sizes to set to the number of files you want to wait for.
    3. Now the merge relation will have a single flowfile containing the file listing. Split them to individual files and process them.

    Have a look at Additional Details of ListSFTP processor. It details how you can wait for your batch to complete process.