We are using spring kafka streams for processing data streams. We are currently using the processing.guarantee: exactly_once configuration to make sure it adheres to Exactly-once semantics of data processing.
We have a new requirement to have some metadata emitted via interceptors. The interceptor uses kafka template to publish metadata (mostly skimming header data of messages) to a kafka topic.
The reason to have it on interceptor is the pipeline to process streams, including multiple applications/topic and it would be easy to wire the interceptors to individual apps.
Question I have is, will the kafka template used in my interceptor require @Transactional annotation or will it use the transaction semantics generated by processing.gaurantee config. And will it be able to rollback data and not commit when primary streams consumer/producer failed to commit?
Any pointers towards this is highly appreciated.
The KafkaTemplate cannot participate in the stream's transaction. It would need a separate transaction, which would break EOS.
For exactly once, either everything has to be done with streams, or everything with spring-kafka.