GitLab API pagination with RestAssured. How to get all pages in one response?

Question

I try to get all issues from GitLab API. GitLab API allows to get issues with pagination, but a maximum of 100 per page:

https://docs.gitlab.com/ee/api/README.html#pagination

As I see I can parse the x-next-page header and return the List<Response> in the result, but it will be better to get all pages in one response.

I found I can use the pagination=keyset parameter but it not helps, I still get 100 issues per page. My attempt:

        return RestAssured.given()
                .log().all()
                .accept("json")
                .header("PRIVATE-TOKEN", TOKEN)
                .queryParam(STATE, "opened")
                .queryParam("pagination", "keyset")
                .queryParam("per_page", "100")
                .when().get(url)
                .then()
                .extract()
                .response();

Answer 1

I'm not familiar with this API but I would conclude from the docs that it's not possible.

pagination=keyset is just a different style of achieving pagination. You are still forced to use pagination.

The default pagination style is offset-based, in which your query says something like "My pages are 100 items long, and I am on page 8". The server then sorts the issues, skips the first 700 (the pages 1-7 that you've already seen - this is the offset), and returns items 700-800.

Using the keyset-based style, your query will say something like "the last item I saw had ID ABCDEF, I want the next 100 items". According to the docs, this way is more efficient, presumably because of the way the server stores the items.

I assume offset-based is the default just because that's the more common way pagination is implemented in other APIs.

---

It's good defensive API design to enforce some upper limits. Not having an upper limit might mean the server has to try to write a huge JSON/XML response for 10 million issues. Lazy developers might use it because it seems "easier", even if they only care about the first few items. It might seem overly restrictive if you know what you are doing ("Well this repo is guaranteed to never have more than 5 issues"), but it is generally much better to assume that every user is potentially malicious.