Is there any way to make my Kafka Stream application automatically read from the newly created topic?
Even if the topic is created while the stream application is already running?
Something like having a wildcard in topic name like this:
KStream<String, String> rawText = builder.stream("topic-input-*");
Right now, I've multiple clients sending data(all with the same schema) to their own topic and my stream application reads from those topics. Then my application does some transformation and writes the result to a single topic.
Although all of the clients could write to the same topic, an unbehaving client could also write on behalf of someone else. So I've created individual topics for each client. The problem is, whenever a new client comes, I create the new topic and set the ACL for them with a script but that is not enough. I also have to stop my streaming application, edit the code, add the new topic, compile it, package it, put it on the server and run it again!
Kafka Streams supports patter subscription:
builder.stream(Pattern.compile("topic-input-*"));
(I hope the syntax is right; not sure from the top of my head... But the point is, instead of passing in a String you can user an overload of the stream() method that takes a pattern.)