Search code examples
apache-kafkakafka-consumer-apirebus

Manually commiting in Rebus with Kafka

Using Kafka as the transport by way of this lib, is there a way with Rebus to give control on when a consumer will commit a message? I want to consume an event from Kafka, do some work which could possibly except/fail, and only if it succeeded tell Kafka to move the offset forward.

Solution

  • I am the author of the library you use "Rebus.Kafka". You can only stop accepting publications completely by unsubscribing from your subscription. Then roll back the offset and re-process the queue. To want to stop processing messages is an unnatural desire for bus users. As an alternative to stopping the bus, you can consider the following options:

    1. Handler pipeline
    2. Automatic retries and error handling
    3. Workers and parallelism

    Please provide a more detailed case that you want to implement.

    Addition:

    If you are not satisfied with the default option EnableAutoCommit = true, you can set its value, as well as all other parameters for the producer and consumer in the configuration parameters. 

    EnableAutoCommit = false,

    How this is done is shown in this example of using transport "Rebus.Kafka": https://github.com/glazkovalex/Rebus.Kafka/blob/master/Examples/Scaleout.Producer/Program.cs

    Notice that the current version of the transport with disabled autoCommit, receives five messages and after processing moves the pointer. In other words, if the service is interrupted with autoCommit disabled, the service can re-process up to five messages when restarting. The number of these re-processed messages can be changed in future versions of the transport "Rebus.Kafka".