Wait for unknown amount of Observables to complete

Question

I have an API I need to pull some data from, but this data is paginated and I have no way to get around it. I don't know how many pages there are, but I can know when I'm at the last one. I want to make a function that gets every page one by one.

This is the page interface:

interface PaginatedResult<T> {
  count: number;
  next: string;
  prev: string;
  results: T[];
}

When I'm at the last page, the next attribute is null, so I essentially want to do something like this:

let data: Data[] = []
let url: string = API_ENDPOINT

while (url!== null) {
  this.http.get(url).subscribe((page: PaginatedResult<Data>) => {
    data = data.concat(page.results)
    url = page.next
  })
}

However, this obviously doesn't work, because it just makes requests at full tilt until one of them sets the url variable to null, and this just freezes the browser. I looked up ways to chain subscriptions together, but I haven't found a way to combine an indeterminate amount of subscriptions together. There is no way to know how many pages are there, I can only get one page at a time and that's all the info I have.

Essentially, the problem is I need to wait for each page to arrive before seeing if I should request the next one.

Any ideas?

As a little note, although this question has a pretty similar title, it doesn't describe the same problem, as in that question the user knows how many requests they need to make, but want to make them in series. In my question I don't know how many requests I have to make, and I want to do them in series.

Answer 1

Not familiar about rxjs but consider doing something like below:

let data: Data[] = []
let url: string = API_ENDPOINT

const handler = (page: PaginatedResult<Data>) => {
  data = data.concat(page.results)
  if (page.next) {
      this.http.get(page.next).subscribe(handler);
  } else {
    // Got everything
  }
}

this.http.get(url).subscribe(handler);

This should get them one at a time sequentially.