Our nifi needs to pull a file produced in splunk, but it appears that the file can end up on any one of the splunk search heads, I don't know if it's possible to always have the file show up on the same searchhead, but it would be difficult to request such a change from the splunk team.
The nifi I inherited solved this by having 4 separate getSFTPs, each one hardcoded to poll one of 4 servers. However, it isn't at all dynamic. I'm now porting this nifi flow to run in a different cloud, and that includes trying to parameterize a number of previously hard coded values. For the getSFTP the problem is that the number of search heads in the new cloud vary depending on if it's dev, int, or prod.
I'm looking for a clean way to rewrite the current getSFTPs to still poll for the file we need, but dynamically do it based off of the number of servers defined in a property passed to it. so if there is only one search head, as in dev, it will only have on getSFTP hitting that one search head, but if there are 4 it knows to poll all four servers in search of the file we need.
Is there an easy way to make this happen?
The way that NiFi works is that in order to pass variables to a processor, that processor must have an upstream connection. That means that there must be a processor that sends data to the desired processor, in order for the desired processor to be able to use variables.
An example is the NiFi Processor PutS3Object, in which one can provide the Object Key value as a variable because the upstream processor can provide the filename. However, with the GetSFTP Processor, the processor doesnt allow for it to connect to an upstream processor. Therefore, one cannot provide a variable to the GetSFTP processor.
To perform the desired action, one cannot use the provided GetSFTP Processor, but change the processor code to perform as desired.