Search code examples
apache-nifi

How to pass attributes to ListHDFS


It seems, the way "list" processor works, we can't put them in middle of a flow. Then how to set attributes to ListHDFS? e.g. if I want to parameterize "directory", and pass it at runtime.


Solution

  • You can use expression language in the Directory property, but only to reference variables, system properties, or dynamic EL expressions, but not flow file attributes.

    ListHDFS (and the other List processors) are made to track state and determine which files are new from the previous listing, so if you were allowed to specify the directory from an incoming flow file this would mean that the directory could change at any moment which then makes the previous state no longer meaningful, or would mean that it would need to track state for N number of directories which could grow large and would be unclear when a directory was no longer being listed.

    It may be helpful to implement another processor that allows dynamic listings, but does not store state,