Increase performance async Parallel.Foreach

Question

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this web app on Azure, I have tested this for 100 it works fine but for 10k it gets timeout issue. Is my implementation in need of any tuning?

private List<Task> taskEventList = new List<Task>();
public async Task ProcessStart()
{
    string messageData = "{\"name\":\"DemoData\",\"no\":\"111\"}";
    RegistryManager registryManager;

    Parallel.ForEach(deviceList, async (device) =>
    {
        // get details for each device and use key to send message
        device = await registryManager.GetDeviceAsync(device.DeviceId);
        SendMessages(device.DeviceId, device.Key, messageData);
    });

    if (taskEventList.Count > 0)
    {
        await Task.WhenAll(taskEventList);
    }
}

private void SendMessages(string deviceId, string Key, string messageData)
{
    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
    //created separate Task
    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));
    taskEventList.Add(taskEvents);
}

private async Task ProcessMessages(string deviceId, string messageData)
{
    var startTime = DateTime.UtcNow;
    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))
    {
        await deviceClient.SendEventAsync(messageData);
    }
}

Answer 1

There's definitely a race condition, at least. Parallel is only for synchronous code, not asynchronous.

As far as I can see, you don't need Parallel or Task.Run (which are both antipatterns for ASP.NET services):

public async Task ProcessStart()
{
  string messageData = "{\"name\":\"DemoData\",\"no\":\"111\"}";
  RegistryManager registryManager;

  var tasks = deviceList.Select(async device =>
  {
    // get details for each device and use key to send message
    device = await registryManager.GetDeviceAsync(device.DeviceId);
    await SendMessagesAsync(device.DeviceId, device.Key, messageData);
  }).ToList();

  await Task.WhenAll(tasks);
}

private async Task SendMessagesAsync(string deviceId, string Key, string messageData)
{
  DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);
  await ProcessMessagesAsync(deviceId, string messageData);
}

private async Task ProcessMessagesAsync(string deviceId, string messageData)
{
  var startTime = DateTime.UtcNow;
  while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))
  {
    await deviceClient.SendEventAsync(messageData);
  }
}

for 10k it got timeout issue.

15 minutes is a long time for an HTTP request. I think it would be worthwhile to take a step back and see if there's a better way to architect the whole system.