Search code examples
apache-kafkayandexclickhouse

Clickhouse is not able to get all data from kafka

I was trying to put some real time metrics from kafka to clickhouse, my query looks like below

CREATE TABLE queue2 (timestamp String, namespace String, data Float, unit String, plugin_running_on String, version UInt64, last_advertised_time String) ENGINE = Kafka('10.224.54.99:9092', 'outtopic1', 'group2', 'JSONEachRow');

as it is getting metrics from kafka for about 150 rows around it suddenly stops and shows the 

156 rows in set. Elapsed: 11.245 sec.

but the data will be still there in kafka which needs to be pushed to clickhouse...what might be the problem? can someone help me out

Solution

  • Do you use MATERIALIZED VIEW for Kafka table as recommended in the documentation?

    If you use MATERIALIZED VIEW then all messages from Kafka will be inserted into the view. So in this case you should select from the view, not from the Kafka table.

    If you do not use MATERIALIZED VIEW then you can query new messages from the kafka table only once. When you query again query will not return the same messages again, because it is already consumed once from Kafka.

    Additionally you can check ClickHouse log (/var/log/clickhouse-server/clickhouse-server.log) for some errors.

    Also make sure that you do not have any other Kafka consumer (or ClickHouse kafka table) with the same group-topic, since within Kafka consumer group message fetched only once.