Javascript generator function with async generator source closure not being maintained?

Question

Given a Javascript generator function that fetches data from an async generator source, and post-processes it, like:

async function* getData() {
  let currentPage = 0
  while(true) {
    let records = await fetch('api/data?page='+currentPage)
    if (records.length == 0) return
    for (record of records) {
      yield record
    }
    currentPage++
  }
}

function* myGenerator() {
  let index = 0
  let gen = getData()
  while(true) {
    index++
    console.log('Setup', index)
    yield gen.next().then(rs => {
      console.log('Processing', index)
      return rc.value.toLowerCase()
    })
  }
}

If it's called multiple times rapidly (faster than the promise factory results take to resolve), the index value gets reported as just the last value in the loop:

let promises = []
let gen = myGenerator()
for (let i = 0; i < 10; i++) {
  promises.push(gen.next().value)
}

Results in "Setup 1, Setup 2, Setup 3, ..., Setup 10", followed by ten repetitions of "Processing 10". The "Setup" calls make sense as it can queue up multiple generator results that are all pending promises, and then once a page of data is available to the async generator, assuming the page of data has 10 or more items in it, all 10 of the pending promises will be resolved rapidly. However, why is it that all of them report the same "10" index value, rather than 1-10? Is there a way within generator functions for this sort of setup to track which original call through the generator the current async response is?

Answer 1

The while loop in myGenerator will complete before any of the promises are fulfilled. This means that index would be 10 by the time any of the console.log('Processing', index) statements are run.

You can fix it with an IIFE like this:

async function* getData() {
  let currentPage = 0
  while(true) {
    let records = currentPage>5 ? [] : [1,2,3].map(i=>String(i+currentPage*10))
    if (records.length == 0) return
    for (let record of records) {
      yield record
    }
    currentPage++
  }
}

function* myGenerator() {
  let index = 0
  let gen = getData()
  while(true) {
    index++
    console.log('Setup', index)
    yield (index=>gen.next().then(rs => {
      console.log('Processing', index)
      return rs.value.toLowerCase()
    }))(index)
  }
}

(async () => {
  let promises = []
  let gen = myGenerator()
  for (let i = 0; i < 10; i++) {
    promises.push(gen.next().value)
  }
  console.log(await Promise.all(promises))
})();

It's easiest to not mix async generators with non-async generators:

async function* getData() {
  let currentPage = 0
  while(true) {
    let records = currentPage>5 ? [] : [1,2,3].map(i=>String(i+currentPage*10))
    if (records.length === 0) return
    for (let record of records) {
      yield record
    }
    currentPage++
  }
}

async function* myGenerator() {
  let index = 0
  let gen = getData()
  while(true) {
    index++
    console.log('Setup', index)
    let v = (await gen.next()).value;
    console.log('Processing', index);
    yield v.toLowerCase()
  }
}

(async () => {
  let promises = []
  let gen = myGenerator()
  for (let i = 0; i < 10; i++) {
    promises.push(gen.next())
  }
  console.log((await Promise.all(promises)).map(r=>r.value))
})();