Parallel.ForEach returning before object's method which makes rate limited API calls

Question

I am working on a plugin for a program that needs to make API calls, I was previously making them all synchronously which, well it worked, was slow.

To combat this I am trying to make the calls asynchronous, I can make 10 per second so I was trying the following:

Parallel.ForEach(
    items.Values,
    new ParallelOptions { MaxDegreeOfParallelism = 10 },
    async item => {
        await item.UpdateMarketData(client, HQOnly.Checked, retainers);
        await Task.Delay(1000);
    }
);

client is an HttpClient object and the rest is used to build the API call or for the stuff done to the result of the API call. Each time item.UpdateMarketData() is called 1 and only 1 API call is made.

This code seems to be finishing very quickly and as I understand it, the program should wait for a Parallel.ForEach() to complete before continuing.

The data that should be set by item.UpdateMarketData() is not being set either. In order to make sure, I have even set MaxDegreeOfParallelism = 1 and the Delay to 3 seconds and it still finished very quickly despite having ~44 items to go though. Any help would be appreciated.

UpdateMarketData() is included below just in case it is relevant:

public async Task UpdateMarketData(TextBox DebugTextBox,HttpClient client,
    bool HQOnly, List<string> retainers)
{
    HttpResponseMessage sellers_result = null;
    try
    {
        sellers_result = await client.GetAsync(String.Format(
            "www.apiImCalling/items/{0}?key=secretapikey", ID));
    }
    catch (Exception e)
    {
        System.Windows.Forms.MessageBox.Show(
            String.Format("{0} Exception caught.", e));
        sellers_result = null;
    }
    var results = JsonConvert.DeserializeObject<RootObjectMB>(
        sellers_result.Content.ReadAsStringAsync().Result);
    int count = 0;
    OnMB = false;
    LowestOnMB = false;
    LowestPrice = int.MaxValue;
    try
    {
        foreach (var x in results.Prices)
        {
            if (x.IsHQ | !(HQOnly && RequireHQ))
            {
                count++;
                if (count == 1)
                {
                    LowestPrice = x.PricePerUnit;
                }
                if (retainers.Contains(x.RetainerName))
                {
                    Retainer = x.RetainerName;
                    OnMB = true;
                    Price = x.PricePerUnit;
                    if (count == 1)
                    {
                        LowestOnMB = true;
                    }
                }
                if (LowestPrice == x.PricePerUnit
                    && x.RetainerName != Retainer)
                {
                    LowestOnMB = false;
                }
            }
        }
    }
    catch (Exception e)
    {
        System.Windows.Forms.MessageBox.Show(
            String.Format("{0} Exception caught.", e));
    }
}

Answer 1

async doesn't work with Parallel. One is asynchronous, the other is parallel, and these are two completely different styles of concurrency.

To restrict the concurrency of asynchronous operations, use SemaphoreSlim. E.g.:

var mutex = new SemaphoreSlim(10);
var tasks = items.Values.Select(item => DoUpdateMarketData(item)).ToList();
await Task.WhenAll(tasks);

async Task DoUpdateMarketData(Item item)
{
  await mutex.WaitAsync();
  try
  {
    await item.UpdateMarketData(client, HQOnly.Checked, retainers);
    await Task.Delay(1000);
  }
  finally { mutex.Release(); }
}

You may find my book helpful; this is covered in recipe 11.5.