IAsyncEnumerable endpoint does not stream data chunk by chunk, instead it waits for all data and then returns

Question

I'm trying to make a WebAPI endpoint in .NET 7.0 that will get streamed text data from OpenAI API and return this data, chunk by chunk, to the user.

Here is my controller:

[HttpGet]
public async IAsyncEnumerable<string> StreamAnswer(string message)
{
    // await foreach (var part in _openAiManager.StreamAnswer(message))
    // {
    //     yield return part;
    // }
    yield return "test";
    Thread.Sleep(3000);
    yield return "test2";
    Thread.Sleep(3000);
    yield return "test3";
    Thread.Sleep(3000);
    yield return "test4";
    Thread.Sleep(3000);
    yield return "test5";
    Thread.Sleep(3000);
}

Here is how I'm trying to receive data in web browser:

const response = await fetch('http://localhost:5159/OpenAi?message=test');
const reader = response.body.getReader();

while (true) {
  const {value, done} = await reader.read();
  if (done) break;
  console.log('Received', value);
}

console.log('Response fully received');

For some reason, I'm always getting all data as one big chunk, just after all code finishes executing in an controller's action. Instead, I want to receive chunks as soon as they arrive.

What am I missing here?

Answer 1

You're blocking the server thread. Instead await Task.Delay(3000);

eg

async IAsyncEnumerable<string> StreamAnswer(string message)
{
    // await foreach (var part in _openAiManager.StreamAnswer(message))
    // {
    //     yield return part;
    // }
    yield return "test";
    await Task.Delay(3000);
    yield return "test2";
    await Task.Delay(3000);
    yield return "test3";
    await Task.Delay(3000);
    yield return "test4";
    await Task.Delay(3000);
    yield return "test5";
    
}

Then running wget from powershell will display the incremental loading of the result.