Can I send output of a Kafka Stream to a REST API endpoint

Question

I currently have an app that consumes from a Kafka topic using Consumer API and publish these messages to a REST API endpoint. I wanted to see if I can use Kafka Streams API do achieve the same. Is that possible?

Answer 1

Technically speaking, yes, you can. It would look something like this (with several details omitted for clarity)

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> inputStream = builder.stream("topic-name");
imputStream.foreach((key, value) -> {
      // Do something with key and value to publish to REST API
                    });

But this isn't what Kafka Streams was designed for; it's meant more for processing records from Kafka, producing the results back to Kafka, and not reaching out to external systems.

The biggest issue I see with this approach is that with Kafka Streams, you don't have direct control over committing. I imagine you'd only want to commit after a bunch of records have been successfully sent to the REST API endpoint, including some retries, etc. So doing something like that is better served with a plain consumer application like you already have.

HTH