multiple httpclient request that depends on data from another httpclient request using observables

Question

I'm re-writing an application in angular 5 that fetch data from a server and download some pdf files. The server is a soap interface with 3 api:

1 - get the list of campaigns

2 - get files array by campaign id

3 - get a file by file id

I've already done this with pure XMLHTTPRequest and some recursive functions that fills a "campaigns" object, extract the cover and saves all the files one after the other.

What i want to get now is to avoid the callback hell and use httpclient and the observables available in angular to obtain the same result, but after two days of study of the rxjs library i still haven't understood how to do it.

I already obtain successfully data with httpclient and i've done lot of try with observable opertors. The schematic flow of events is:

get campaigns xml =>

transform xml data to an array of campaigns (done with map operator) =>

for each campaign get the filesInfo =>

transform xml data to an array of filesInfo =>

for each file of each campaign get that file =>

extract the cover of the file and save the file.

i'm stuck here:

getCampaigns():Observable<Campaign[]>{
    return this.getRawDataProvider.getRawData('campaignsList').pipe(
      map(rawCampaigns => {return this.createCampaignArray(rawCampaigns)})
    )
  }

this.getCampaigns().subscribe(result => {
  console.log(result) //correctly shows an array of campaigns (without the files of course)
})

what now? should I use "from" operator to create new observable from the array? Or a foreach on the array and start a new http client request for each campaign? At the end i need an array of campaigns like this:

[
  {
    name: 'campaign name',
    id: 1234,
    files:[
      {
        name:'file name',
        id: 4321,
        cover: 'base64string'
      }
    ]
  }
]

and i need that the files is downloaded one after another and not in parallel

edit: thanks also to dmcgrandle answer i figured out sone things and after several attempt i came to this:

updateCampaigns():Observable<Campaign[]>{
    let updatedCampaigns:Campaign[];
    return this.getRawDataProvider.getRawData( 'campaignList' ).pipe(
      map( rawCampaigns => {
        updatedCampaigns = this.createCampaignsArray( rawCampaigns );
        return updatedCampaigns;
      } ),
      switchMap( _campaigns => from( _campaigns ).pipe(
        concatMap( _campaign => this.getRawDataProvider.getRawData( 'campaignDocuments', _campaign.id ) )
      ) ),
      map((rawDocuments,i)=>{
        updatedCampaigns[ i ].documents = this.createDocumentsArray( rawDocuments );
        return;
      } ),
      reduce( () => {
        let { campaigns, documentsToDowloadIdList } = this.mergeWithStoredData( updatedCampaigns );
        updatedCampaigns = campaigns;
        return documentsToDowloadIdList;
      } ),
      switchMap( list => from( list ).pipe(
        concatMap( documentId => this.getRawDataProvider.getRawData( 'document', documentId ).pipe(
          map( b64file => { return { documentId:documentId, b64file:b64file } } )
        ) )
      ) ),
      concatMap( file => this.saveFile( file.b64file, file.fileId ) ),
      tap( fileInfo => {
        updatedCampaigns.forEach( campaign => { campaign.documents.forEach( document => {
          if( fileInfo.fileId == document.id ) {
            document.cover = fileInfo.b64cover;
            document.numPages = fileInfo.numPages;
          }
        } ) } )
      } ),
      reduce( () => {
        this.storageservice.set( 'campaigns' , updatedCampaigns );
        return updatedCampaigns;
      } )
    )
  }

Note that i've modified the saveFile function, now it return an observable. Now, this works... I'm wondering if there's a simpler way and if I'm missing something. I am also trying to handle errors with catchError operator, but if i use throw, the type of the observable becomes void and I have to put the type "any" instead "Campaign[]" to make it work. Last thing, i would prefer this observable to return the campaigns object every time it is updated with a new document, so as to display the covers when the file is downloaded and not all together at the end... it's possible to do that?

Answer 1

I suggest you do this all in a single function. Once you have the array of campaigns, emit each item one at a time and do the transformations on each item, one emission at a time. Then finally put the results all into an array and return that from the function.

During development I would also insert tap()s at each stage to see what is going on after each transformation to make sure it's functioning as expected.

Using pseudo-code, it would look something like this:

getCovers(): Observable<Cover[]> {
  return this.getRawDataProvider.getRawData('campaignsList').pipe(
    map(rawCampaigns => this.createCampaignArray(rawCampaigns)),
    concatAll(), // emit the array one item at a time
    concatMap(campaign => this.getFileInfo(campaign)), // get FileInfo for each campaign
    concatMap(fileInfo => this.getTheFile(fileInfo)), // get the File for each fileInfo
    concatMap(file => this.saveTheFileAndGetCover(file)), // save the file, return the cover
    toArray() // put the results into an array for returning.
  )
}