Search code examples
apache-kafkakafka-producer-apireactive-kafka

Reactive kafka producer


Am exploring reactive kafka and just wanted to confirm if reactive kafka is equivalent to sync producer. With sync producer, We get message delivery guarantee with ACK all and producer sequence is maintained. But, delivery and sequencing is not guaranteed with ASYNC. Is reactive producer equivalent to SYNC or ASYNC?


Solution

  • Reactive means async. In plain Kafka clients API also, the KafkaProducer is asynchronous. It becomes synchronous when you explicitly call the kafkaProducer.send().get() which blocks the execution of the program.

    Even with async producer, message delivery is guaranteed. It depends on the no. of retries and the delivery.timeout.ms.

    With ack=all it is ensured that data is replicated among the ISR and you get fault-tolerance with consistency guarantees, in case if one of the broker dies, the consumers see what is intended to be seen.

    As for the sequence, messages are batched before they are sent. If multiple batches are sent asynnchronously like batch-1, batch-2, batch-3 and if for some reason batch-1 fails after batch-2 and batch-3 have been sent, then retying batch-1 will produce batch-1 after batch-2, batch-3, thereby making it out of sequence.

    If you want sequence, you need to ensure that max.in.flight.requests.per.connection set to 1 so that only one request can be in-flight at any given instant of time per producer. However, this can have performance impact. You may want to tune other settings like batch.size like increasing it for example, to get increased throughput with the inflight requests set to 1.

    So your assumption that delivery and sequencing is not guaranteed with ASYNC is false.