Search code examples
restazure-data-factoryazure-synapse

Azure Data Factory: Repeating First Page while Fetching Data from API Pagination

I am facing an issue while using Azure Data Factory to fetch data from an API that supports pagination. It successfully reads and fetches all the data from the first page, but when it tries to move on to the next page, it repeats the process for page one again and again till the request failed. The hasMore in the metadata is True till the pages end in the API. The details are given in the screenshot.

API metadata

As a result, I'm unable to retrieve the complete data from the API. I have checked my pipeline configuration and the pagination settings, but everything seems to be in order. I suspect there might be an issue with the way Azure Data Factory handles pagination or with my configuration. I would appreciate any insights or suggestions on how to resolve this problem.

I have used the Pagination rules EndCondition in Azure data factory. The EndCondition we have provided is as $hasMore=false then the API should end and we should get all the data till the end page. But it does not move to the next page.

ADF Pagination Rule implemented

Solution

  • We must use EndCondition if the variables are in AbsoluteUrl ,QueryParameters or Headers the end variable is not pre-defined, and the end condition depends on the Response of API.

    You must first paginate the URL using the AbsoluteUrl/QueryParameters/Headers with the Range or nexpagelink URL.

    • Next, set the following end condition rule:
    "EndCondition:$.meta.page.hasmore":"Const:False"

    insert description of image here

    when the value of a particular node in the response is a user-defined const value, the pagination should be stopped.

    Visit this MS Document for more details.