kafka kafka-consumer-groups.sh --describe returns no output for a consumer group

Question

kafka version 1.1

--list can get the consumers group

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list --command-config config/client_security.properties
Note: This will not show information about old Zookeeper-based consumers.

spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0
console-consumer-23379
console-consumer-44846
console-consumer-75221
console-consumer-57833
yizhisec-traffic
console-consumer-91940
group1

can't get any information. This consumer group id is generated by spark.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0  --command-config config/client_security.properties

get nothing

Note: This will not show information about old Zookeeper-based consumers.

But test other consumers work.

update more detail

---

I use Structured Streaming kafka

and when I use zookeeper to query get error

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0  --command-config config/client_security.properties

get error

Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
Error: The consumer group 'spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0' does not exist.

Answer 1

kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --describe \
  --group your_consumer_group_name

will return no output if the consumers in the consumer group have not committed any offsets.

---

If the output of

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 \
  --command-config config/client_security.properties

and

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --new-consumer
  --describe \
  --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 \
  --command-config config/client_security.properties

matches and the consumer group appears in both consumer groups list output, my guess is that your consumers inside the consumer group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 have not committed any offsets and this is why no information appears using the describe command.

---

Make sure that your consumers inside spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 commit their offsets succesfully (either manually or automatically).