Azure EventHubs inizialization performances and WebApi2

Question

I have a WebApi2 controller that receives XmlHttpRequests from JavaScript.

I have +500 calls to the api per second, and any request perform some quick calculations, then I create an Azure Storage Queue (not the service bus one) passing in a serialized object for later processing. Until here all works, the problem is that 10-15% of the time, just initializing the Storage queue and adding a 20k JSON message takes something between 500ms to 2 seconds. I sharded the requests into 10 different queues but the problem remains and seems not to be related to amount of traffic, basically sometimes the queues just get sort of stuck into the creation and slow down.

I disabled Nagle and Expect100Continue already.

I thought to convert this architecture in using EventHUbs since probably my situation is requiring an ingestor of events more than a simple Queue, requiring max speed.

But the inizialization of the EventHub has the same exact problem! It takes sometimes 2 or 3 seconds to start and receive a single message, with an average of 400ms.

I measured the speed with a stopwatch.

This is my code in the API Controller:

  var eventHubClient = StorageHelpers.InitializeEventHub("name", "Send");
                           await eventHubClient.SendAsync(new EventData(Encoding.UTF8.GetBytes(QueueSerialized)));

Where InizializeEventHub is:

        public static EventHubClient InitializeEventHub(string eventHubName, string type)
    {
        string connectionString = RoleEnvironment.GetConfigurationSettingValue("Hub"+type+eventHubName);
        return EventHubClient.CreateFromConnectionString(connectionString, eventHubName);}

The service is hosted on azure using a cloud service, hosted in the same place (WestUS) of the ServiceBus and storages.

My questions are:

• 1)Is this amount of time normal to inizialize the connection?
• 2)Is there a way for Web Api to share the same EventHubClient instance for all calls? Something like is done with Redis using ConnectionMultiplexer in a Lazy class.
• 3) May I cache the EventHubClient Object?

Any help on this matter would be really appreciated, I can even return on the Storage Queue if there is some way to speed the initialization and the AddMessageAsync operation.

Thanks

Answer 1

I ended up with a crazy easy solution. Both EventHubs and StorageQueues needs time to initialize and EventHubs in particular very often is slow in adding messages to the stream. Now 300ms is not slow in 99.99% of cases but in my case it is.

StorageQueue is super cheap, fast and easy but slow as hell adding messages. After hours of benchmarks, and other solutions checks like Redis Pub/Sub, I ended up using StorageQueues, simply not awaiting the Async call.

so the standard call is

await queue.AddMessageAsync(message);

and the await part is the problem, WebApi cannot return if the task is not back. Should be a Fire and Forget but it is not.

i resolved the matter not awaiting the call, hiding the warning using a variable

var nowait = queue.AddMessageAsync(message);

The insert in the queue is -immediate- in any situation, and no messages are lost.