Can't paginate using Http Client snap Post method in Snaplogic

Question

Please help!! I am extracting data using POST HTTP Client snap with a body filtering the data by the datetime. I am having an issue with pagination... I do not get the next url, I just get the next increment value. e.g. with limit of 10 starting with 0 the next increment will be after 10 then next will be after 20 etc... How do I use that to paginate. How do I configure the HTTP client snap to loop through the data and re-execute the url to retrieve the next group of records?

I have tried this pipeline where the execute pipeline snap calls a pipeline that just has the httpclient snap and ingests the after value with the next incremented value. the first row processes 0-100; 2nd row 100-200; and 3rd row 200-300

How can I make this loop to process all 400,000 records?

Answer 1

I was able to accomplish what I wanted by changing the pipeline.

I essentially used a solution provided here https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3097822264/Pagination with a few tweaks. I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap "after" : $splitValue, "limit" : 100 Then I was getting subsequent data. But the data was duplicating. So I modified the mapper for pagination as follows:

sl.range(1, parseInt($entity.total), 100)

I also changed the 1st http snap to just give me 1 record just and I used the results to get the total and I process from page 1 to the end in the rest of the pipeline so I don't do a union.

And it worked! I am now getting subsequent data and it the records are not duplicating.

Note: there is a HubSpot 'search' endpoint limitation that it can only return up to 10000 records so anything more than that cannot be retrieved using this method.