Search code examples
apache-kafkamicroservicesdapr

Dapr pub/sub is slow

I have created a small performace-test application with C#

  • with Confluent Kafka.Net client and
  • with Dapr using a pubsub componennt

Using a simple topic with 1 partition hosted on my local machine, I can get up around 1000 messages/sec sustained sending rate with the confluent library.

With Dapr i only get around 80-100 messages/sec.

With Dapr :

await client. PublishEventAsync("kafka-pubsub-noauth", "topic1", msg);
ProducedMessages++;

With confluent


in the configration : acks=1
// dont know where to set this for Dapr, 0 is a little faster, 
// "all" is the same as 1 as i only have one broker 

producer.Produce("topic1", msg, KafkaDeliveryHandler);
....

private void KafkaDeliveryHandler(
  DeliveryReport<string, CustomMessage> deliveryReport)
{
...
ProducedMessages += 1;
...
}

//Wait at the end to get the DeliveryReports for all produced messages.

sending several thousand messages, taking the time it took, and then dividing the time/messagecount.

I have switched the pubsub.yaml component now also to Redis and RabbitMQ, and the performance of the Dapr PublishEventAsync seem to be the bottleneck, as I also reach max 100-150 messages/sec.

Any ideas how to improve this ?

Solution

  • It turns out that the PublishEventAsync call is indeed a bottleneck if you want to send more than a couple hundred messages per second.

    But it turns out that the Dapr team is already working on a Bulk-Publish API https://github.com/dapr/docs/blob/12bbac91d47c5753e34e9dd84bda9dabeae02e66/daprdocs/content/en/reference/api/pubsub_api.md#publish-multiple-messages-to-a-given-topic which gives much better performance.

    I was able to increase it to 9000 messages/sec with a batch-size of 100 (on a fast Azure-VM, locally it only goes up to ~1000 m/sec like the confluent library)

    Here is also a helpful sample I got from Shubham Sharma https://github.com/shubham1172/sample-dapr-bulk-publish

    For now you will need to compile your own version from the main branch if you want to use the new API. After compiling you need to replace daprd(.exe) in your .dapr directory with the new version.