Search code examples
apache-kafkakafkajs

Kafka Producer: Disconnect after sending message vs keeping connection open


I've not been able to find an answer to this in the kafkajs docs or from skimming through the official Apache Kafka design docs, but in their producer examples, the producer disconnects after sending the messages. However, that could be because it's a trivial example, and not a long running process.

For long running applications, like web apps, I'm wondering if it is better to disconnect from the producer after sending messages, or if it is better to (presumably) keep the connection open throughout the life of the running application.

An obvious advantage to keeping the connection open is that it wouldn't to reconnect when sending messages, and an obvious disadvantage would be that it holds a TCP connection open. I don't know how big of an advantage or disadvantage either are.

My guess would be that it depends on the expected volume; if the application is going to be constantly sending messages, it'd be best to keep the connection open, while if it is not going to be sending messages frequently, it would be appropriate to disconnect after sending messages.

Is this an accurate assessment? I'm more wondering if there are nuances that I've missed or made incorrect assumptions.


Solution

  • It is recommended to have producer open for the scope of the application. Only if you have it open you can leverage the properties like batch.size and linger.ms to improve the throughput of your application.

    Even for less busy applications it's better to have a producer instance shared in your application.

    However if you're looking to enable transactions, you might want to consider implementing a pool of producer instances.

    Although KafkaProducer is thread-safe, it does not support multiple concurrent transactions, so if you want to run different transactions concurrently, it's recommended to have separate producer instances.