Search code examples
javaapache-kafkapartitionconsumer

consumer.How to specify partition to read? [kafka]

I am learning Kafka and I want to know how to specify the partition when I consume messages from a topic.

I have found several pictures like this:

enter image description here

It means that a consumer can consume messages from several partitions but a partition can only be read by a single consumer (within a consumer group).

Also, I have read several examples for consumer and they look like this:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "consumer-tutorial");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

and:

Subscribe:

consumer.subscribe(Arrays.asList(“foo”, “bar”));

Poll

 try {
      while (running) {
        ConsumerRecords<String, String> records = consumer.poll(1000);
        for (ConsumerRecord<String, String> record : records)
            System.out.println(record.offset() + ": " + record.value());
      }
 } finally {
      consumer.close();
 }

How does this work? From which partition will I read messages?

Solution

  • There are two ways to tell what topic/partitions you want to consume: KafkaConsumer#assign() (you specify the partition you want and the offset where you begin) and subscribe (you join a consumer group, and partition/offset will be dynamically assigned by group coordinator depending of consumers in the same consumer group, and may change during runtime)

    In both case, you need to poll to receive data.

    See https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html, especially paragraphs Consumer Groups and Topic Subscriptions and Manual Partition Assignment