We're investigating options to optimizie our disk space usage. While doing that we were looking at reserved disk space by kafka partitions. Right now the error topics use the same number of partitions as the origin topic. We do not really require partitioned error topics and kafka allocates a good amount of space for each partition.
So the question is, is it possible to set a lower partition count for the dlq using the recently introduced configuration dlqProducerProperties?
When we hacked some lower partition count into the code we were facing some issue that the messages could not be moved to the dlq because the destination partition would not match. I guess the origin message's partition index is reused to move it into the dlq destination (to the same partition index), but it only has 1 instead of 9 partitions. Is that a limitation of the kafka client or spring cloud stream?
The binder currently maps partitions 1:1 to the DLQ topic; this is to retain order (for those users that care).
In the Spring for Apache Kafka project (used by the binder), the DeadLetterPublishingRecoverer has a configurable destination resolver which allows sending dead-letters to a different partition in the DLQ topic.
However, this is not currently used by the binder (where dead-lettering pre-dated the support in the core project).
Please open a new feature GitHub issue against the binder.