Getting pages results from rest API: Best practice

Fairly new to rest API and have only done some fairly basic queries to get what I need so far.

Writing a new PowerShell script that pulls out all the devices using rest:

$url = "https://my-site.com/internal/api"

Then querying all of the devices and assigning those results to a variable so I can then go off and do things with that variable later in different points throughout my script:

$devices = Invoke-RestMethod -uri "$url/devices" -UseDefaultCredentials -Method Get -ContentType "application/json"

There are over ten thousand results, and what I want to do is be able to search the results across all returned pages.

So far I've worked out I can do something like this to get all the results in one page:

$devicespagetest = Invoke-RestMethod -uri "$url/devices?pagesize=99999" -UseDefaultCredentials -Method Get -ContentType "application/json"

Is this bad practice?

Is there a more efficient or programmatic way to achieve this without doing it as I have above?

I'm not actually sure what variety of the Rest API we're using, but I know we don't integrate it with Jira or Confluence.

Solution

Is the API you are calling something developed by yourself?

Edit: just saw that you are calling WordPress API in your tags. WordPress offers pagination links, and provides useful pagination information through custom headers.

Usually, REST APIs provide pagination support when returning list of resources through query parameters (e.g.: "offset" and "max"). The offset parameter indicates how many results should be skipped and max (or pagesize in your case) determines the maximum of results you want to fetch.

For example, http://myapi.com/persons?_offset=0&_max=20 represents the first 20 persons in your list. In the context of the HATEOAS constraint, it is recommended to provide HTTP Link response headers so that clients know how to navigate through the pages using the “previous”, “first”, “next”, and “last” relationships (standardized in RFC 5988).

For example:

GET /persons?_offset=30&_max=10 HTTP/1.1
Host: myapi.com
Accept: application/json

Results in the following response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 934
Link: <http://myapi.com/persons?_offset=0&_max=10>; rel="first"
Link: <http://myapi.com/persons?_offset=20&_max=10>; rel="previous"
Link: <http://myapi.com/persons?_offset=40&_max=10>; rel="next"
{body}

In order to protect the server for clients requesting the whole list resource (i.e. without specifying the pagination parameters), the following approach is recommended. When a client requests a list resource (e.g. http://myapi.com/persons), the server redirects the client to the first X (e.g. 10) items.

The corresponding HTTP conversation is shown below:

GET /persons HTTP/1.1
Host: myapi.com
Accept: application/json

Results in the following response:

HTTP/1.1 303 See Other
Location: http://myapi.com/persons?_offset=0&_max=10

Next, the client follows the redirect:

GET /persons?_offset=0&_max=10 HTTP/1.1
Host: myapi.com
Accept: application/json

This results in the following response:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 850
Link: <http://myapi.com/persons?_offset=40&_max=10>; rel="next"
Link: <http://myapi.com/persons?_offset=120&_max=10>; rel="last"
{body}

Note, however that redirection can be a cause of problems in environments where you need CORS (Cross Origin Resource Sharing). In that case the representation of the partial list retrieved by GET /persons can simply indicate how many persons were retrieved and how many there are in total (optional).

If your API provides such navigation links, it is probably better to use them and implement a loop with Invoke-RESTMethod and build indexes for your search on the fly instead of fetching 10000 results at once, which will put unnecessary pressure on your server. But without knowing your use case, it is difficult to give you a proper advise.