We are using confluent platform on ubuntu. We have simple JSON data being sent through a cURL request to kafka-rest server on kafka topic named "UE_Context".
A kafka stream named "UE_CONTEXT_STREAM" is created for this topic with the below command:
CREATE STREAM UE_Context_Stream (ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', VALUE_FORMAT='JSON');
A kafka table named "UE_CONTEXT_TABLE" is created for this topic with the below command:
CREATE TABLE UE_Context_Table ( registertime BIGINT, ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', KEY='ue_key', VALUE_FORMAT='JSON');
I have two rows of data being pumped on topic using the below cURL commands:
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x1234", "ecgi" : "1234"}}]}' "http://localhost:8082/topics/UE_Context"
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x4321", "ecgi" : "4321"}}]}' "http://localhost:8082/topics/UE_Context"
I have a select query waiting on table as below:
This query displays the table info when JSON data is pumped into the topic. We then stop pumping JSON data into the topic and end the select query and end the select query. If a select is performed at a later point of time, the previously populated table info is not displayed. Is there no way to persist this data? Kafka connectors and using a DB might be an option. But does kSQL not have transient memory to store the table info?
a select is performed at a later point of time, the previously populated table info is not displayed.
A select statement defaults to the latest offsets of the topic
If you want to see previous data, you need to set the consumer offset back to the beginning
SET 'auto.offset.reset'='earliest';
Also, as mentioned in the documentation (with emphasis)
A SELECT statement by itself is a non-persistent continuous query. The result of a SELECT statement isn’t persisted in a Kafka topic and is only printed in the KSQL console. Don’t confuse persistent queries created by CREATE STREAM AS SELECT with the streaming query result from a SELECT statement.