Search code examples
apache-nifi

NiFi - Call Rest API for every row in the file

I have a datset of IDs, I've got a flow file that has one row per ID. I have an API that takes this ID as a parameter, and I want to harvest the results for all rows back into NiFi (example below).

https://service.com/api/thing/{ID}

How in NiFi, can I call this API, for all IDs in my dataset. Ideally using some parallelism if possible.

(for reference, in SSIS I could load these IDs into an array and then loop over an API call with a parameter for the ID).

Solution

  • First, use SplitText to get each Id as a flowfile enter image description here

    Then copy content to an attribute by ExtractText . add custom property such as 'message.body' in this example

    enter image description here

    so that ExtractText would add message.body.0 attribute to the flowfile and you can use it InvokeHttp like below . Please note that since your endpoint is https , you may need to configure SSL Contect Service

    enter image description here

    Finally , you can set concurent task count for each Processor for parallelism

    enter image description here