Search code examples
messagingapache-kafka

Guaranteeing delivery with clustered Kafka

According to a recent article, Kafka can drop acknowledged messages under a replicated configuration:

https://aphyr.com/posts/293-call-me-maybe-kafka

In case one must be absolutely certain that a message was delivered to a consumer, will this solution work in every case:

  • establishing two topics, one for sending, one for receiving
  • producer sends a message, including a token (an UUID) on the sending topic
  • producer saves the sent message and the token to local storage, independently from Kafka
  • producer erases the sent message from local storage only when it receives a confirmation message from the consumer on the receiving topic, containing the same token that was sent on the sending topic.
  • producer resends message periodically, with the same token, until it receives a confirmation. The unique token ensures that even if the message is received twice (the first being not confirmed for any reason) it is identified as duplicate by the consumer.

Does it work even in case of dropped messages? Is there a name for this pattern?

Solution

  • That seems a bit over-engineered. It's enough to keep track of offsets on the consumer side and either commit offset or not, depending on what you consider to be a successful message processing by the downstream system.
    The split-brain problem that Aphyr found was fixed (I think in 0.8.2.0) with unclean.leader.election.enable setting, which you need to set to false (default is true, since it's a last-resort measure).

    You should also look at ack producer configuration setting, which determines how many in-sync replicas need to acknowledge write in order to be considered successful.

    You can read more about these settings and general delivery guarantees Kafka provides here, near the bottom of the page.

    PS It might be worth tracking this issue.

    Hope that helps :)