Search code examples
apache-kafkaksqldb

How to transform a nested array into a table in ksqldb/kafka

I started learning ksqldb for Kafka and ran into a problem, I have a Product stream and the structure is the following:

 ID          | VARCHAR(STRING)
 NAME        | VARCHAR(STRING)
 ...
 INGREDIENTS | ARRAY<STRUCT<NAME VARCHAR(STRING), ISALLERGEN BOOLEAN, RELATIVEAMOUNT DOUBLE>>

And as a result, I want to create a table and get the last state of the product using 'LATEST_BY_OFFSET' function, but the problem is that I can't apply this function to Array or Struct.

Should I EXPLODE each prop and create a separate entry in the table for each ingredient?(I think it's very strange). How can I deal with this situation? Can you tell me if you have any ideas?

Solution

  • The problem was solved in release 0.25.1 https://github.com/confluentinc/ksql/pull/8878 https://github.com/confluentinc/ksql/commit/adac45855dce1c413073e5cbeb474cb022013a2b