Search code examples
javascripttypescriptasync-awaitaxioses6-promise

Using Promise to get paginated response using axios

I am trying to figure out how one can use Promise and async/ await to fetch the data from a rest API that returns the data with some pagination logic.

Specifically I am trying to fetch Azure Users data using MS Graph API. Since azure does not send all the data related to Users in a single API, we have to make multiple calls to build the data for a single Azure User.

To give some insights into my code/logic, here is how I am fetching the Users' basic data using the pagination logic:

let nextPageLink: string| null = null;
do {
      let pagedResult: AggregationPageResult = await this.fetchNextPage(nextPageLink);
      let usersMap: Map<string, any>[] = pagedResult.data!;
      nextPageLink = pagedResult.nextPageLink!;
      this.sendPageResult(usersMap, res);
} while(nextPageLink)

The pagination logic here is simple and doesn't bother me. Let us say the first call to the Users Graph API returns 100 users. Now, for each of these 100 Users, I have to make separate calls to get their Group Membership data, Directory Roles Data, Last Login Activity and other 'N' number of details.

Here is the code that I am using to get the Group Membership details for each User in current page:

for(let user of usersToSearch) {
    let groupMembershipAPI = "https://graph.microsoft.com/v1.0/users/" + user + "/memberOf/microsoft.graph.group";
    
     let httpConfig: HttpRequestConfig = {
             baseURL: groupMembershipAPI, 
             method: 'GET',
             queryFilter: filters
         }
    
      executions.push(httpClient.executeRequest(httpConfig, true));
}    

let executionResults: HttpResponse [] = await Promise.all(executions);

Now this code works fine and get Group Membership details for all the Users, iff the memberships are less, or in other words the membership data fits to the default page size or any page size for that matter. What if the data is more, what if any particular User has 1000 group memberships and the default page size is only 100. In that case, my current logic will only bring 100 memberships since I am not executing it for next page

For all the requests that I am pushing to executions I also want to take care if the API is sending a valid nextPageLink with it, so that the code goes to fetch the membership data for that User again till the nextPageLink is empty.

Now this is just one scenario for Group Memberships, consider what will happen if I want to bring all the other data like Directory Roles, PIM data, Last Login Activity and more!

The original pagination logic used to bring User Page can be employed here as well, but I am looking for a solution where I can recursively or dynamically add Promise chains or is there any other mechanism that can help me to solve it in a better way? I also want to make sure regarding performance and resource utilisation

Edit 1: My intention to use a different approach for pagination while fetching user properties such as its group memberships/ roles etc is because:

  1. I want to process User pages sequentially and synchronously
  2. On the contrary, while fetching these these properties, they can run in parallel.
  3. Meaning, what I want to achieve is that, fetching User group membership properties can run in parallel with fetching User Roles...
  4. I don't want to wait to start Role fetch because User Group fetch is in progress. Both can run in parallel
  5. If I use the original approach, then that will be sequential.

Pseudo code to explain my above points:

 async public fetchNextPage(nextPageLink: string): AggregationPageResult {
    let propertyFetchResults = []
     1. await fetch the next user page
     2. for every extra property to be fetched with pagination
          2.a propertyFetchResults.push (...result)
     3. await Promise.all(propertyFetchResults)
     4. merge the above property results with users of current page
    
 }
Solution
    1. First you need to understand do you really need to fetch all pages with do/while? For most of cases you need to fetch the next page only when user scrolled to the bottom or clicked next page.

    Also, make sure you need to get all additional data when loading user pages. If you have user list and user details screen, you may not need all the data on the list screen, and you can fetch additional data only when opening user details.

    1. Aggregate ids first if possible (for example get all group ids of all users on the page), put them to Set to remove duplicates and then fetch. Use API that accepts array instead of one id, for example getGroups(ids) instead of multiple getGroup(id):
    const fetchUsers = async (page = 1) => {
  const users = await fetch(...)

  const promises = []

  // fetch groups
  const groups = []
  const groupIds = Array.from(new Set(users.map(x => x.groupId)))
  while (groupIds.length) {
    const idsToFetch = groupIds.splice(0, 100) // Splice is used here if 100 is a limit of ids for this API
    promises.push(fetchGroups(idsToFetch).then(x => groups.push(...x)))
  }

  // fetch roles
  const roles = []
  const roleIds = Array.from(new Set(users.map(x => x.roleId)))
  while (roleIds.length) {
    const idsToFetch = roleIds.splice(0, 100) // Splice is used here if 100 is a limit of ids for this API. Makes sense only if there are more than 100 roles.
    promises.push(fetchRoles(idsToFetch).then(x => roles.push(...x)))
  }

  ...

  await Promise.all(promises)

  return {
    users,
    groups,
    roles,
  }
}

    If APIs with array ids parameters are not available, use your pagination method with do/while to load all data.

    1. Remember about error handling. What if one request fails? Do you want to cancel all other requests with AbortController, or retry several times failed requests, or ignore etc?

    2. Remember about caching - you might have already some entities in your cache, and you may skip fetching them again if they are fresh enough.

    3. Don't merge entities (denormalize) together without a good reason. In most cases it is better to cache data separately, normalized, and access later when needed by id:

    const group = store.getState().entities.groups[user.groupId]

    It will keep your state consistent, with only one instance of each entity. And when it updates in the cache after next fetch, it will update in the whole app UI instead of just one particular screen.

    But if you think that normalization is too much for now, and it probably is, feel free to skip it and just merge data while fetching.

    1. Axios is not a must have lib, most of devs can't even explain why they use it.