Kafka creating more partitions than hardcoded in the properties file

Question

I have configured my producer as :

spring.cloud.stream.bindings.pc-abc-out-0.destination=pc-abc-my-topic
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=5
spring.cloud.stream.bindings.pc-abc-out-0.producer.header-mode=headers
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=10
spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

However, in the kafka logs , I keep getting the error :

o.s.kafka.support.LoggingProducerListener - Exception thrown when sending a message with key='byte[14]' and payload='byte[253]' to *topic pc-abc-my-topic and partition 8*: org.apache.kafka.common.errors.TimeoutException: Topic pc-abc-my-topic not present in metadata after 60000 ms.

The highlight in the error message being : topic pc-abc-my-topic and partition 8

Why is it looking for partition 8 even though I have defined number of partitions as 5. Shouldn't the numbering be betweenn 0-4. I have several other error messages with partition number more than 5.

Earlier in my configuration I had added

spring.cloud.stream.kafka.binder.auto-add-partitions=true

But I removed it and we scaled down and scaled up the services. The issue still exists. Is this the case of stale config?

Answer 1

The issue turned out to be this configuration:

spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

The partition number being calculated because of payload.key is somwhow greater than the partitionCount value we configured and created on our infrastructure. Removing these two configuration stopped the issue.