Slow response sending data to IoT Hub using Device Client

I'm using a .NET web app as a buffer between an IoT device and IoT Hub. The IoT device sends data to the web app which then routes that data to Azure IoT Hub. I'm using the Device Client to do this. It worked fine until the number of requests started growing as we went into production.

Here is the bit of code that sends data to IoT Hub.

public async Task SendDataToIoTHub(Message eventMessage, string deviceId, string deviceKey)
{
    string connectionString = $"HostName=" + IoTHubHostname + ";DeviceId=" + deviceId + ";SharedAccessKey=" + deviceKey;
    using (DeviceClient deviceClient = DeviceClient.CreateFromConnectionString(connectionString))
    {
        if (deviceClient == null)
        {
            return;
        }

        await deviceClient.SendEventAsync(eventMessage);
    }               
        
}

I have about 3k-4k requests every minute and the above code gets called for each request. It has really started being unreliable since the request rate hit this amount. There was a ' too many TCP connections' warning for a brief period of time which caused the entire application to start slowing down dramatically.

I'm wondering if this initialization of the device client is correct or if there is a better way to do it.

Thanks!

Solution

This could be due to reaching the maximum outbound TCP connections. The limits are:

1,920 connections per B1/S1/P1 instance
3,968 connections per B2/S2/P2 instance
8,064 connections per B3/S3/P3 instance
16,000 connections per I1/I2/I3 instance

Source (This post also shows how to view the number of TCP connections over time)

You're opening a new connection every time you create a new DeviceClient, so with those rates, you will hit some limits. You could scale up your instance, but maybe you should consider keeping a single DeviceClient per device (if you don't have thousands of devices).

There might be a reason for this "web app as a buffer" as you call it, and if you want to continue using it, you might consider using the REST API to send your device events instead. You don't need an AMQP connection if you're only going to send one event.