Kafka Streams QueryableStore - ReadOnlyKeyValueStore fast access

Question

I have the following setup. Kafka-Broker running on kubernetes. One SpringBoot-Application with kafka-streams and one input-topic as KSteram.

On the stream I group on the key and use aggregation and build a MaterializedView -> KeyValueStore RocksDB. The DB includes 135048 key/value pairs.

I read all key/value-pairs like this

private ReadOnlyKeyValueStore<String, PriceDomain> keyValueStore;

public List<PriceModel> fetchAllPriceInErrorState() {

  if (this.keyValueStore == null) {
            this.keyValueStore = queryService.getQueryableStore(
                     ModelStrings.PRICE_STORE_NAME,
                     QueryableStoreTypes.keyValueStore()
             );
         }

  while (keyValueIterator.hasNext()) {
    KeyValue<String, PriceDomain> keyValue = keyValueIterator.next();
    PriceDomain priceDomain1 = keyValue.value;
    PriceModel priceModel = convertToPriceModel(priceDomain1);
    priceModelList.add(priceModel);
  }
}

This will take ~3-4 sec to execute. Is there a way to increase the execution-time to fetch all key-value-pairs from the store? Sure, most time i can access the value directly over the key and this will be fast. But sometimes I need all key/value-pairs.

Say there are much more key/value-pairs in the future (~500-1000k) then this will take ~ 5-10 times longer

thanks for help

Answer 1

I think you have to set higher number of partitions (input topic based on which store is build) and than run several such applications. Each instance will only consume some subset of messages (Key -> Values) and it will run faster