I have a topic with many partitions. I want many applications read all messages from this topic. Some applications read frequently and other at midnight.
I don't find any help, for this problem in stackoverflow or in book.
How i can implement that in kafka ?
Thanks
You'd need Kafka Consumers in order to let your applications consume messages from Kafka topics. In case each of your application needs to consume all messages, then you'd have to assign a distinct group-id to every application.
Kafka assigns the partitions of a topic to the consumer in a group. It also guarantees that a message is only ever read by a single consumer in the group. Partitions is the level of parallelism in Kafka (if you have 2 partitions then you can have up to 2 consumers in the same consumer group).
Example:
Say you have topic example-topic with 5 partitions and 2 applications each of which must consume all the messages from example-topic. You will need two distinct consumer groups (1 per application), say group.id=app-group-1 for the first app and group.id=app-group-2 for your second app. Within each consumer group, you can start at most 5 consumers consuming messages from your topic. Therefore, up to 5 consumers will subscribe to topic example-topic and belong to group.id=app-group-1 and another 5 consumers that will subscribe to topic example-topic and belong to group.id=app-group-2.