I have recently started studying Apache Kafka.
I configured zookeeper , Also configured Kafka instance(broker).
Everything is working fine.
Yesterday i sent lot of messages using producer(using default partition) and consumed them .This activity created lot of log folders with format
Topic-Name Partition-Number eg:- Ajinkya-0 , Ajinkya-10 , Ajinkya-12 etc...
Today when i restarted Apache Kafka , i got lot of logs
[2018-10-27 15:09:19,917] INFO [Log partition=__consumer_offsets-39, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2018-10-27 15:09:19,917] INFO [Log partition=__consumer_offsets-39, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2018-10-27 15:09:19,918] INFO [Log partition=__consumer_offsets-39, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 2 ms (kafka.log.Log)
[2018-10-27 15:09:19,919] INFO [Log partition=__consumer_offsets-21, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Recovering unflushed segment 0 (kafka.log.Log)
[2018-10-27 15:09:19,919] INFO [Log partition=__consumer_offsets-21, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2018-10-27 15:09:19,920] INFO [Log partition=__consumer_offsets-21, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2018-10-27 15:09:19,920] INFO [Log partition=__consumer_offsets-21, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 1 ms (kafka.log.Log)
[2018-10-27 15:09:19,922] INFO [Log partition=Ajinkya-74, dir=/home/ajinkya/software/Kaftka/kafka_2.11-2.0.0/Kafka-logs] Recovering unflushed segment 0 (kafka.log.Log)
Why do i see logs as
Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 1 ms (kafka.log.Log)
Since message has already been consumed , why do i still see loading logs ?
Also i saw loading logs for all partitions i.e 50 partitions
As no of partitions will increase , no of logs will also increase .
Kafka is not like traditional messaging middleware (JMS, RabbitMQ etc).
Records are retained in the logs for 7 days (by default). See log.retention.hours and log.retention.minutes. Consumers can "rewind" back as far as the retention period.
I removed the spring-kafka tag since this question has nothing at all to do with Spring.