Bypassing record limit in API Call using Powershell

Question

I'm using Powershell to fetch a list of locations from a vendor site, using an API call:

# Set the initial page and limit values
$page = 1
$limit = 1000

# Initialize an array to hold all the data
$allData = @()

# Set the headers with the token key
$headers = @{
    'Content-Type' = 'application/json'
    'Authorization' = 'Bearer SomeTokenKey'
    'Accept'= 'application/json'
}

# Loop until all data is retrieved
while ($true) {
    # Set the API endpoint URL with the current page and limit values
    $uri = "https://mysite.Someapisite.com/api/locations?page=$page&limit=$limit"

    # Make the API call and retrieve the data
    $responseData = Invoke-RestMethod -Uri $uri -Method GET -Headers $headers -Body $body

    # Add the retrieved data to the allData array
    $allData += $responseData.data

    # Check if all data has been retrieved
    if ($responseData.meta.pagination.current_page -eq $responseData.meta.pagination.total_pages) {
        break
    }

    # Set the page and limit values for the next API call
    $page = $responseData.meta.pagination.current_page + 1
    $limit = $responseData.meta.pagination.per_page
}

# Export all the data to a CSV file
$allData | Export-Csv -Path 'c:\extracts\locations.csv' -NoTypeInformation -Force

The vendor has a limitation of extracting 1000 records max per call. The vendor API documentation states that pagination is supported:

The site utilizes an offset/limit based pagination via the limit and page parameters where the offset is automatically calculated based on the parameters provided.

Link: <https://mySite.someAPIsite.com/api/location?page=3&limit=10>; rel="next", 
      <https://mySite.someAPIsite.com/api/location?page=7&limit=10>; rel="last"

My goal is to extract all records at source, but the above script only returns 1000 records.

(A) Have I not implemented this correctly?

(B) Will using pagination bypass the record limit?

Any thoughts?

Answer 1

I played around the code, and got it do to what I needed:

# API endpoint
$apiUrl = 'https://someSite.siteUrl.com/api/locations'

# API credentials
$accessToken = 'InsertTokenHere'
      
# Set the limit and initial page
$limit = 1000
$page = 1

# Initialize the locations array
$locations = @()

do {
    # Construct the API request URL with pagination parameters
    $url = '{0}?limit={1}&page={2}&expand=locationHours' -f $apiUrl, $limit, $page

    try {
        # Invoke the REST API to fetch locations
        $response = Invoke-RestMethod -Uri $url -Headers @{ Authorization = "Bearer $accessToken" }

        # Add the fetched locations to the locations array
        $locations += $response.data

        # Increment the page number for the next API request
        $page++
    }
    catch {
        # Output the error message and break out of the loop
        Write-Host "API request failed with error: $_"
        break
    }
    
} until ($response.data.Count -ne $limit)

if ($locations) {
    # Output the fetched locations to a CSV file
    $locations | Export-Csv -Path "locations.csv" -NoTypeInformation

    # Output success message with record count
    $recordCount = $locations.Count
    Write-Host "Successfully fetched and saved $recordCount records to locations.csv."
} else {
    Write-Host "No Records were fetched."
}