Kafka consumer not consuming from beginning

Question

I have Kafka setup on my local machine and have started the zookeeper and a single broker server.

Now i have a single topic with following description:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic edu-topic --describe
Topic:edu-topic PartitionCount:3    ReplicationFactor:1 Configs:
    Topic: edu-topic    Partition: 0    Leader: 0   Replicas: 0 Isr: 0
    Topic: edu-topic    Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: edu-topic    Partition: 2    Leader: 0   Replicas: 0 Isr: 0

I have a producer which have produced some message before the consumer was started as follows:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic edu-topic
>book 
>pen 
>pencil
>marker
>

and when i started the consumer with --from-beginning option, it does not shows all the messages produced by the producer:

~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning

However, it is showing the newly added messages.

What's wrong i am doing here? Any help?

Answer 1

--from-beginning: If the consumer does not already have an established offset to consume from, start with the earliest message present in the log rather than the latest message.

Kafka consumer uses --from-beginning very first time if you retry which I suspect you did, it will start from where it left. You can consume the message again with any of the below options

1. reset consumer group offset using below

kafka-streams-application-reset.sh --application-id edu-service --input-topics edu-topic --bootstrap-servers localhost:9092 --zookeeper 127.0.0.1:2181

then retry again from the beginning

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning

2. Use new consumer id which will start consuming from staring points

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group new-edu-service --from-beginning

3. You can also use an offset instead to consume the next N messages from a partition

kafka-console-consumer.sh --bootstrap-server localhost:9092 --offset 0 --partition 0 --topic edu-topic

--offset <String: consume offset> : The offset id to consume from (a non- negative number), or 'earliest' which means from beginning, or 'latest' which means from end (default: latest)
--partition <Integer: partition> : The partition to consume from Consumption starts from the end of the partition unless '--offset' is specified.