Search code examples
spring-cloudspring-cloud-streamspring-cloud-dataflow

Spring cloud stream - Kafka binder performance

I have the following simple piece of code:

    private int i = 0;

    @StreamListener(Sink.INPUT)
    public void processMessage(Message<?> message) {
         i++;
    }

    @Scheduled(fixedDelay=5000)
    private void scheduled(){
        LOG.info("Messages consumed: " + i);
    }

and the following properties:

spring.cloud.stream.bindings.input.consumer.headerMode=raw
spring.cloud.stream.kafka.binder.autoCreateTopics=false
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOffset=false
spring.cloud.stream.bindings.input.destination=test6
spring.cloud.stream.bindings.input.group=testGroup50
spring.cloud.stream.bindings.input.partitioned=false

I have a local kafka topic with a single partition with 96 k messages. The simple kafka consumer provided by the kafka library consumes those messages in roughly 4 seconds.

However, the above code takes close to 1 minute!

enter image description here

Obviously, it is a concern for our application, has anyone experienced this before? Am I missing something here?

Visual VM isn't flagging anything either.

PS: I just tried with auto commit and I am still seeing an atrocious performance.

Solution

  • Thanks to @Marius Bogoevici's research, it was discovered that the poor performance was caused by Mac OS:

    https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/71#issuecomment-263280685

    Posting this for reference in case anyone else encounters this issue.