Pagination in Zapier

Question

I am trying following code to get all records from a paginated API in Zapier.

const limitPerPage = 20;
const apiUrl = "https://myurl.com/data";
var lastCursor = null;
var output = null;

const getContent = async function (cursor) {
    let actualUrl = apiUrl + `?cursor=${cursor}&limit=${limitPerPage}`;
    var apiResults = await fetch(actualUrl)
        .then(resp => {
            return resp.json;
        });
}

const getEntireContentList = async function (cursor) {
    const results = await getContent(cursor);
    console.log("Retreiving data from API for cursor : " + cursor);
    if (results.metadata.cursor !== "") {
        return results.concat(await getEntireContentList(results.metadata.cursor));
    } else {
        return results;
    }
};


(async() => {
    const entireList = await getEntireContentList();
    console.log(entireList);
    output = entireList;
    callback(null, entireList);
})();

I get error as You did not define output! Try output = {id: 1, hello: await Promise.resolve("world")};

How can I fix this?

Answer 1

Your problem is that though you're awaiting in that function, the top-level carries on and execution ends before your code has had a chance to run.

The good news is, Zapier wraps your code in an async function already, so you can use await at the top level (per these docs).

Try this instead:

const limitPerPage = 20;
const apiUrl = "https://myurl.com/data";
let lastCursor = null;
// var output = null; // zapier does this for you already

const getContent = async function (cursor) {
    const actualUrl = apiUrl + `?cursor=${cursor}&limit=${limitPerPage}`;
    const rawResponse = await fetch(actualUrl)
    return resp.json() // async function, you had it as a property
}

const getEntireContentList = async function (cursor) {
    const results = await getContent(cursor);
    console.log("Retreiving data from API for cursor : " + cursor);
    if (results.metadata.cursor !== "") {
        return results.concat(await getEntireUserList(results.metadata.cursor)); // should this be named getEntireContentList?
    } else {
        return results;
    }
};


return {
  results: await getEntireContentList()
}

I noticed this is a recursive approach. That's fine, but remember that you've got limited execution time. You also might hit memory limits (depending on how many objects you're returning), so keep an eye on that.