Search code examples
spring-bootapache-kafkaspring-kafka

Have I need to define transaction id to enable idempotence producer?

I found in spring-kafka to enable idempotence producer I have to set transactionIdPrefix, but I don't need transactions, only idempotence.

 public void setTransactionIdPrefix(String transactionIdPrefix) {
    Assert.notNull(transactionIdPrefix, "'transactionIdPrefix' cannot be null");
    this.transactionIdPrefix = transactionIdPrefix;
    enableIdempotentBehaviour();
}

So I must to define transaction id or idempotence still will be working on retry ack level of default kafka-client?

Solution

  • You do NOT need to enable transactions to enable idempotence; that code is there because idempotence is required for transactions; the user must not have configured it to be false.

    See the boot documentation for how to set arbitrary Kafka properties:

    https://docs.spring.io/spring-boot/docs/current/reference/html/messaging.html#messaging.kafka.additional-properties

    The properties supported by auto configuration are shown in the “Integration Properties” section of the Appendix. Note that, for the most part, these properties (hyphenated or camelCase) map directly to the Apache Kafka dotted properties. See the Apache Kafka documentation for details.

    Properties that don’t include a client type (producer, consumer, admin, or streams) in their name are considered to be common and apply to all clients. Most of these common properties can be overridden for one or more of the client types, if needed.

    Apache Kafka designates properties with an importance of HIGH, MEDIUM, or LOW. Spring Boot auto-configuration supports all HIGH importance properties, some selected MEDIUM and LOW properties, and any properties that do not have a default value.

    Only a subset of the properties supported by Kafka are available directly through the KafkaProperties class. If you wish to configure the individual client types with additional properties that are not directly supported, use the following properties:

    spring:
  kafka:
    properties:
      "[prop.one]": "first"
    admin:
      properties:
        "[prop.two]": "second"
    consumer:
      properties:
        "[prop.three]": "third"
    producer:
      properties:
        "[prop.four]": "fourth"
    streams:
      properties:
        "[prop.five]": "fifth"

    or

    spring.kafka.properties[prop.one]=first
spring.kafka.admin.properties[prop.two]=second
spring.kafka.consumer.properties[prop.three]=third
spring.kafka.producer.properties[prop.four]=fourth
spring.kafka.streams.properties[prop.five]=fifth

    This sets the common prop.one Kafka property to first (applies to producers, consumers, admins, and streams), the prop.two admin property to second, the prop.three consumer property to third, the prop.four producer property to fourth and the prop.five streams property to fifth.

    In any case, it is enabled by default, if you haven't changed the related properties; see https://kafka.apache.org/documentation/#producerconfigs_enable.idempotence