how to find out the produce order acked in the librdkafka library?

Question

in librdkafka,

The data has been put in the transmission queue via following function.

virtual ErrorCode   produce (Topic *topic, int32_t partition, int msgflags, void *payload, size_t len, const void *key, size_t key_len, void *msg_opaque)

The result of the transfer can be found in the dr_cb() registered as a callback through the poll(0).

class ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb {
 public:
  void dr_cb (RdKafka::Message &message) {
    // I want to know the order of produce.
  }
};

If I produce 10 times, how can I know if the third produce has been successful? I can see the actual messages or the number of messages left in the transmission queue, but the result of third produce is unknown.

I want to synchronize the sequence number of the message produced and the sequence number of the message reported to be complete via dr_cb. What should I do?

Answer 1

Ordering is only a thing per partition in Kafka, if you produce to multiple partitions there is no ordering inbetween those partitions.

As for messages produced to the same partition they will be produced in their original order, unless there is an error which warrants a retry, in which case reordering is possible.

You can either set max.in.flight=1 to avoid reordering, which unfortunately also decreases throughput and increases latency, or use the Idempotent Producer with enable.idempotence=true to get guaranteed ordered delivery at very little throughput cost.

You can use the message opaque to attach a pointer to a message in produce() which you'll get back as msg_opaque in the delivery report, allowing you to map delivery reports to the original object you produced.