Search code examples
apache-kafkakafka-consumer-apikafka-topic

Do each Kafka Producer/Consumer occupy a one partition in topic?


I have a question of Kafka topic. I made system like below. And I made a Kafka topic (1 partition) using below command.

enter image description here

./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic temp.t

And then, I met errors like below.

2023-06-09 12:03:14,420 [kafka-producer-network-thread | producer-1] ERROR org.springframework.kafka.support.LoggingProducerListener: Exception thrown when sending a message with key='temp.t' and payload='{"aaa...' to topic temp.t: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for temp.t-0:1010 ms has passed since batch creation

I thought It may be happened by topic's partition, so I re-created topic(3 partitions) using below command.

./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic temp.t

After that, First errors were not happened. In this situation, I have a question. Are each Kafka's producers and consumers occupying the topic's each partitions? I think, the reason why first errors is producer lost occupied partition by consumer.

Thank you read my question even though my English is bad, and please give me advices of my question.

Thank you.

changed topic's partition number


Solution

  • Your error is simply saying you are waiting too long to flush the producer.

    It's unrelated to topic or consumer settings, and producers know nothing about consumers.