Search code examples
xmlpdfapache-nifi

Nifi - Moving a pair of files between two folders

Good day

I am fairly new to Apache Nifi and I have my difficulties with some basic logic (although I once worked as software dev) within Nifi. As far as I understood, Nifi's flow files are always one "physical" file (payload/content) with its meta data (attributes).

How can I guarantee, that a file pair (two files belonging to each other) get copied to another location in a defined order?

The situation is the following:

  • System A generates an unknown amount of "paired files" (consisting of 1x XML and 1x PDF with the same name) in Folder X. The PDF files need a tic longer to be generated.
  • Nifi should take the files from System A/Folder X and first move the PDF to System B/Folder Y
  • after guaranteeing the the PDF has reached its destination, move the XML to System B/Folder Y

System B has a folder listener which further processes the xml+pdf and will cause errors if the xml is there before the pdf - this part cant be changed

My current approach is not working, I understand why, but I cant find a solution on how to solve my problem with the corresponding files.

  1. ListFile (filtered on XML)
  2. FetchFile (${absolute.path}/${filename})
  3. FetchFile (${absolute.path}/${filename:substringBeforeLast('.'):append('.pdf')})
    • if it fails, retry X times with RetryFlowFile
  4. RouteOnAttribute (${filename:toLower():endsWith('pdf')}) <-- here it fails, file is xml
  5. PutFile (pdf)
    • Abort if it fails
  6. FetchFile (${absolute.path}/${filename})
  7. RouteOnAttribute (${filename:toLower():endsWith('xml')})
  8. PutFile (xml)

Long story short. I believe I am doing this completely wrong and I am in need of some advice (not the solution ;) )

Solution

  • Why are you fetching the xml file first ? If you want to move your files only when the pdf is here, you just have to check when the pdf appears, and then move your files.

    You could do the following :

    • Getfile (to get the pdf when it appears in the Folder X)
    • Putfile (to put the pdf into the Folder Y)
    • UpdateAttribute (to update the filename from .pdf to .xml)
    • FetchFile to move the xml file to Folder Y.