Kafka Streams DSL folding hierarchical data

Question

Using Kafka Streams DSL, this is what I want to do:

Input message Serdes: Avro for both key and value Key: Record with fields L1, L2, L3 Value: Record with value V (in this case an int)

What I want to do is collapse this heirarchy in such a way that the produced stream has the correct summed up value. For example,

Input:
L1   L2   L3  V
a1   b1   c1  v1
a1   b1   c2  v2
a1   b2   c1  v3
a1   b2   c2  v4
a2   b1   c1  v5



Output 1: (Data wanted at L1, L2)
L1   L2   V
a1   b1   v1 + v2
a1   b2   v3 + v4
a2   b1   v5

Output 2 (Data wanted at L1)
L1    V
a1    v1 + v2 + v3 + v4
a2    v2

Is there a way the Streams DSL would allow be this? Note that the key type changes across all outputs and I couldn't find a way to perform these rekey + aggregation (since rekey is esentially supposed to merge multiple values). While there might be ways to achieve this using the processor API or basic Kafka Consumer, want to check how to do this in DSL (if possible).

Answer 1

You should be able to use selectKey():

KStream input = builder.stream(...);
input.selectKey(/*create a new output record with only 2 attributes L1 and L2*/)
     .groupyByKey()
     .aggregate(...);

input.selectKey(/*create a new output record with only 1 attribute L1*/)
     .groupByKey()
     .aggregate(...)