How after completing any task in the loop, pass a value to another method, return to the loop, and wait for the next task to complete?

Question

There are 2 methods, one method depends on the other. I need to send several files to the server in parallel, which answer from the server comes first, and send it to DownloadFileAsync method, and do the same with the rest (without waiting for all answers at once, send them (answers from the server) to DownloadFileAsync() method, as they are received).

public async Task CompressAllFilesAsync(List<UserFile> files, string outputpath)
{
    await UploadAllFilesAsync(files);
    await DownloadAllFilesAsync(files, outputpath);
}

public async Task UploadAllFilesAsync(List<UserFile> files)
{
    IEnumerable<Task> allTasks = files.Select(async (file, i) =>
        files[i].FileLink = await UploadFileAsync(files[i])
    );

    await Task.WhenAll(allTasks);
}

public async Task DownloadAllFilesAsync(List<UserFile> files, string outputpath)
{
    IEnumerable<Task> allTasks = files.Select((file, i) =>
        DownloadFileAsync(files[i].FileLink,
        Path.Combine(outputpath, $"{files[i].FileName}")
    ));

    await Task.WhenAll(allTasks);
}

Now the program is waiting for all answers from the server (links to downloadable files) before running the DownloadAllFilesAsync() method.

Answer 1

You could consider using the new (.NET 6) API Parallel.ForEachAsync, and execute the UploadFileAsync and then the DownloadFileAsync sequentially for each UserFile, but concurrently with other files:

public async Task CompressAllFilesAsync(List<UserFile> files, string outputPath)
{
    var options = new ParallelOptions() { MaxDegreeOfParallelism = 10 };
    await Parallel.ForEachAsync(files, options, async (file, ct) =>
    {
        await UploadFileAsync(file);
        await DownloadFileAsync(file.FileLink,
            Path.Combine(outputPath, $"{file.FileName}"));
    });
}

The UploadFileAsync and DownloadFileAsync will be invoked on ThreadPool threads. This shouldn't be a problem, unless those methods interact with thread-affine components (like UI controls).