As far as i understand the best way to organise broadcast(i.e. every consumer receives all messages) is to give each consumer it's own consumer group-id.
The problem is that "If all consumers in a group leave the group, the group is automatically destroyed"
(source: https://jaceklaskowski.gitbooks.io/apache-kafka/kafka-properties-group-id.html)
so that means that if my consumer goes down the corresponding entry with the key groupId,topicName,partitionNumber to __consumer_offsets
will be removed, meaning that when the consumer goes up again, with the same group-id it wont be able to read messages that were sent while it was down.
Does anyone know a solution to this?
the group is automatically destroyed doesn't mean, that whole information about group disappear. I think it relates to data, that are kept in memory. Information about offset are not removed from __consumer_offsets. Depending on value of offsets.retention.minutes property, (broker property) old offsets are removed. By default 7 days (10080 minutes)
In Apache Kafka documentation you can find information about offsets.retention.minutes property brokers configs
offsets.retention.minutes - After a consumer group loses all its consumers (i.e. becomes empty) its offsets will be kept for this retention period before getting discarded. For standalone consumers (using manual assignment), offsets will be expired after the time of last commit plus this retention period.
It means, that if for offsets.retention.minutes minutes none of consumer from particular group will be connected, information about offset will be deleted.