am trying to use storm HDFS spout, I have Apache Nifi moving files to HDFS directory where storm is listening, but as soon as apache nifi starts to move the file, storm senses that and starts processing noting that the file haven't been completely moved.
I have tried to use conf.put(Configs.IGNORE_SUFFIX, ignoreSuffix) from storm side
and apache nifi updateAttribute to rename the file to .ignore
I need to rename the file again after being completely moved, how can I achieve that ? or is there another way ?
When writing to HDFS, NiFi will write the file with the filename containing a dot at the beginning like ".foo.txt" and when the write operation is complete it will rename it to the name without the dot like "foo.txt". So if Storm has a way to ignore a prefix then you should be able to ignore anything starting with a dot.