How to limit calls to Blob Storage from EventProcessorHost?

Question

I have two Azure Eventhubs, each with 32 partitions. Around 700 events pass through the eventhubs every second, first through one, then through the other. One EventHub has 3 concurrent readers/EventProcessorHost, and the other only has one.

My issue is, that the C# EventProcessorHost seems to make a pretty wild amount of calls to the blob storage, for things like lease management and checkpointing. We're talking on average, 800 calls every second.

We register the EventProcessorHost like this

            _eventProcessorHost = new EventProcessorHost(
                _eventHubOptions.CurrentValue.EventHubName,
                _eventHubOptions.CurrentValue.ConsumerGroupName ?? PartitionReceiver.DefaultConsumerGroupName,
                _eventHubOptions.CurrentValue.EventHubConnectionString,
                _eventHubOptions.CurrentValue.StorageConnectionString,
                _eventHubOptions.CurrentValue.StorageName);

            // Reduce lease renewal throughput for storage accounts to save price
            _eventProcessorHost.PartitionManagerOptions = new PartitionManagerOptions()
            {
                LeaseDuration = TimeSpan.FromSeconds(60),
                RenewInterval = TimeSpan.FromSeconds(50)
            };

            var eventProcessorOptions = new EventProcessorOptions
            {
                InitialOffsetProvider = partitionId =>
                    EventPosition.FromEnd(),
                EnableReceiverRuntimeMetric = true,
                MaxBatchSize = 400,
                PrefetchCount = 400,
            };

            await _eventProcessorHost.RegisterEventProcessorFactoryAsync(_eventProcessorFactory, eventProcessorOptions);

We're looking to limit the amount of calls made to storage. As you can see above, we've also extended the lease duration, from the default of 30 seconds, to the maximum of 60.

We've implemented it, so that checkpointing happens every 2000 messages, which means that, on average, checkpointing happens no more than every 15 seconds per partition.

Is this a normal amount of storage calls to coordinate the EventProcessorHosts? Is there any other dials or knobs I can turn, e.g. adjusting partitions to limit the amount of storage calls?

Answer 1

Given that you have already optimized processor host and partition manager options, there are only couple things I can recommend:

1. Reduce number of partitions. Single partition can process up to 1MB/sec or 1K events/sec. Unless of course consumers are doing heavy compute during handling the events.

2. Reduce number of hosts if hosts are running idle. This will help to reduce Storage I/O overhead originated from each host.