Search code examples
spring-bootapache-kafkaspring-kafka

Spring Kafka Listener to pause processing of messages for certain period of time

Here is my use case.

  1. I have an end-of-the-day feed process that runs on a separate process on a scheduled basis.

  2. During the day, I will receive real-time updates through Kafka topics which the Spring Kafka listeners will consume and process. But, I want to pause the Kafka listeners during the EOD feed job kick-started and resume once the job completes.

Here are the approaches I am considering.

  1. I already exposed JMX beans to stop and start the listeners for some admin purposes. The same I want to leverage to stop/start the Kafka listeners when the EOD feed job kicks in. This will be more controlled synchronously.

  2. As I have more control over the EOD job when that kicks in and when that will complete. During that time, the Kafka listeners can be paused based on the schedule. As this is asynchronous, this will cause problems for some reason the EOD feed job takes more time to complete than usual.

Are there any other options to stop the Kafka listeners for a particular amount of time?

Solution

  • Assuming you are using @KafkaListener, you can pause/resume (or stop/start) listener containers using the KafkaListenerEndpointRegistry bean.

    Give each listener an id to get it from the registry.

    See @KafkaListener Lifecycle Management in the documentation.

    If you are not using @KafkaListener and using listener containers directly as beans, you can simply pause/resume, or stop/start the container beans themselves.