Search code examples
restpaginationazure-data-factory

Can REST response body be compared to dynamic content in EndCondition pagination rule in Azure Data Factory?

Is it possible to use the EndCondition pagination rule to compare a REST response body value to dynamic content, like a value from an earlier Lookup activity?

I have successfully created pipelines that paginate until an empty response like [] is returned, which is defined using the pagination rule below:

"paginationRules": {"EndCondition:headers.Content-Length": "Const:2"}

Is it possible to use pagination rules to paginate until the response body matches a dynamic content value? For example, I have tried many versions of this code (which does not work):

"paginationRules": {"EndCondition:body.data.id": "Constant:@activity('LookupLastID').output.firstRow.LastID"}

The primary problem I'm facing is that my REST source only lets me specify a sort_order query parameter, so without an end condition I will end up querying the API for all results every time which is not feasible. I'm interested in any solution that can solve this in Azure Data Factory, even if it does not involve using pagination rules.

I have tried various passes at referencing dynamic content in the EndCondition pagination rule but none were successful.

Solution

  • Is it possible to use the EndCondition pagination rule to compare a REST response body value to dynamic content, like a value from an earlier Lookup activity?

    Yes, it is possible to compare value from earlier Lookup activity in EndCondition pagination rule.

    I tried with my sample Rest URL and to compare value from earlier Lookup activity in EndCondition you need to specify it as dynamic value like below:

    @concat('Const:',activity('Lookup1').output.value[0].page)

    enter image description here

    So, it will check the value of lookup output with EndCodition value and proceed pagination.