Rust Multi-thread chaining HTTP requests and returning results in Iterator

I have an response that returns a list of items backed by cursor-based pagination, that goes like this:

{
  "object": "collection",
  "pages": {
    "per_page": 500,
    "next_url": "https://api.com/v2/assignments?page_after_id=80469434",
    "previous_url": null
  },
  "total_count": 523,
  "data_updated_at": "2017-09-21T11:45:01.691388Z",
  "data": [
    {
      "id": 80469400,
      "value": "owo",
    }
    // ...
  ]
}

Here I have a next_url set here, and there are 500 results per page (the total count is 523) so I can get 500 results now and make a second request to that url for the remaining results.

But I'd like to get the first results now, and not wait for all the requests to finish.

My idea was something like:

spin up one new thread to make all requests in the background (only one, since to get the next url, we need the previous request
use a single producer single consumer queue to put in the results as I get them
on the user side, return some kind of iterator that fetches the results, and blocks if the queue is empty

Would this work? Are there other easier ways to do this? I'm not sure on the last step in particular.

Solution

Yes this would work just fine, however I'd personally implement this with async rather than with multiple threads. No need to spin up a whole thread whose whole job is just waiting around most of the time, and you can still keep your queue setup.

As far the last step, I think the iterator approach would be fine, there's many ways of approaching it and that's a perfectly valid way.