Search code examples
apache-nifi

How to handle REST API Pagenation with Apache NIFI


There is RESTAPI endpoint to query for the documents modified over a period of time. As the the period could be large, the response for that query has a field like number-of-pages, current-page-numer. It will also return 10 documents detail in that response. Basically the page size is fixed at 10.

In python code, I could write a code to loop through all the page numbers to send multi requests to get all the documents' detail, since I know the number-of-pages and it won't change given the same date range.

How to do that in NIFI? If there is a sample nifi template, that will be even better.


Solution

  • It's a loop and setting attributes

    Example here

    https://github.com/tspannhw/FLiPN-AirQuality-REST/blob/main/AirQuality%20(1).json