How to build Observable from dynamically created and destroyed sources?

Question

My question is similar to this one RXJS: Single Observable from dynamically created Observables (no answer there).

I want to parse some pages continuously. There is the main page where I can get a list of links of pages to parse. This list of links changes over time. Then I follow the links and parse the pages continuously as well until their link disappears from the main page and start parsing new pages as they appear on the main page.

My setup for now is as follows. I have a class that given a url can return an Observable of continuously emitting items (parses a single link from the main page). This works great. However, I want to have a "master" class that will be able to return an Observable of same items but taken from multiple pages. The problem that I have is the list of pages is changing (and so is the list of underlying Observables and I can't just use Observable.merge.

TL;DR: I have multiple Observables that I want to merge. But this list of Observables is changing dynamically and I don't know how to handle this.

How can I approach this?

Answer 1

If you already have Observable of "main" page, and a function to fetch items based on this data. You can use switchMap operator for "switching" this dynamically changing list, something like:

getMainPages().pipe(
  switchMap(main => getItemsFromMultiplePages(main))
)

where:

• getMainPages() - return Observable of main page data
• getItemsFromMultiplePages(main) - return Observable of items, created by combining(maybe using merge) data from multiple pages