How to use await in a parallel foreach?

Question

So I spent the better part of the night trying to figure this out.

I was fortunate to get introduced to Parallel.ForEach yesterday and it works like I want it to do except from one detail.

I have the following:

Parallel.ForEach(data, (d) =>
{
    try
    {
        MyMethod(d, measurements);
    }
    catch (Exception e)
    {
        // log
    }
});

Within the method "MyMethod" I have a lot of logic that gets done and most of it is fine but I make API calls where I fetch data and I use an async task for this to be able to use await in order for the code to wait until that specific part gets executed and then move on:

private async void MyMethod(PimData pimData, IEnumerable<ProductMeasurements> measurements)
{
    try
    {
        // a lot of logic but most relevant part 

        await Task.WhenAll(ExecuteMeasurmentAndChartLogic(pimData.ProductNumber, entity));

        await Task.WhenAll(resourceImportManager.HandleEntityImageFiles(pimData.ProductType + pimData.ProductSize,SwepImageType.Png, ResourceFileTypes.ThreeD, entity, LinkTypeId.ProductResource));

        await Task.WhenAll(resourceImportManager.HandleEntityImageFiles(pimData.ProductSketch, SwepImageType.Png, ResourceFileTypes.Sketch, entity, LinkTypeId.ProductResource));

    }
    catch (Exception e)
    {
        // log
    }
}

Problems:

1 For starters the loop finishes before all code is finished

2 Second problem is that I get "Task was cancelled" in a lot of API calls

3 And third as mentioned above, the code does not wait for each method to fully execute.

I cant get it to execute everything in ExecuteMeasurmentAndChartLogic() method before moving forward to the next step.

This gives me the following issues (more issues):

In this method I create an item and add it to the db, and this item needs more info that I get from an API call that is done inside of ExecuteMeasurmentAndChartLogic() but the problem is that several items get created and have to wait for the rest of the data which is not what I desire.

SIDE-NOTE: I am aware that creating an item and adding to the db before all data is there is not best practice but I am integrating toward PIM and the process for that is delicate

I want several threads running but at the same time I want the full logic to execute for each item before moving on to the next method.

Clarification:

Several items running

Each item handles ALL the logic it needs to handle before moving on to the next part of the code. Normally do this with await.

In the code above resourceImportManager() method gets executed before ExecuteMeasurmentAndChartLogic() is finished, which is what I don't want.

Instead of a Parallel.ForEach I used :

Task task1 = Task.Factory.StartNew(() => MyMethod(data, measurements));
Task.WaitAll(task1);

But that wasn't much of a help.

Fairly new to this and haven't been able to understand where I am doing it wrong.

EDIT: Updated the problems with this

EDIT: this is how ExecuteMeasurmentAndChartLogic() looks:

public async Task ExecuteMeasurmentAndChartLogic(string productNumber, Entity entity)
{
    try
    {
        GrafGeneratorManager grafManager = new GrafGeneratorManager();
        var graphMeasurmentList = await MeasurmentHandler.GetMeasurments(productNumber);

        if (graphMeasurmentList.Count == 0) return;

        var chart = await grafManager.GenerateChart(500, 950, SystemColors.Window, ChartColorPalette.EarthTones,
                    "legend", graphMeasurmentList);

        await AddChartsAndAddToXpc(chart, entity, productNumber);
    }
    catch (Exception e)
    {
        Console.WriteLine(e);
    }
}

EDIT: Background to this: I make a call to an API to get a lot of data. For each item in this data I need to make an API call and get data that I apply to the item.

After reading comments which also got me thinking in a different way. I can perhaps loop through all my items and do minor logic for them and add a URL in a task list and make a separate task that executes this one by one.

Will keep this updated

Answer 1

Don't use Parralel.ForEach at all. Make your method to return Task instead of void, collect all the task and wait them like:

Task.WaitAll(data.Select(d => MyMethod(d, someParam)).ToArray());