Difference between kafka idempotent and transactional producer setup?
When setting up a kafka producer to use idempotent behaviour, and transactional behaviour:
I understand that for idempotency we set:
enable.idempotence=true
and that by changing this one flag on our producer, we are guaranteed exactly-once event delivery?
and for transactions, we must go further and set the transaction.id=<some value>
but by setting this value, it also sets idempotence to true?
Also, by setting one or both of the above to true, the producer will also set acks=all.
With the above should I be able to add 'exactly once delivery' by simply changing the enable idempotency setting? If i wanted to go further and enable transactional support, On the Consumer side, I would only need to change their setting, isolation.level=read_committed? Does this image reflect how to setup the producer in terms of EOS?
Yes you understood the main concepts.
By enabling idempotence, the producer automatically sets acks to all and guarantees message delivery for the lifetime of the Producer instance.
By enabling transactions, the producer automatically enables idempotence (and acks=all). Transactions allow to group produce requests and offset commits and ensure all or nothing gets committed to Kafka.
When using transactions, you can configure if consumers should only see records from committed transactions by setting isolation.level to read_committed, otherwise by default they see all records including from discarded transactions.