Search code examples
apache-kafkaapache-kafka-streams

Kafka Streams use case

I am building a simple application which does below in order -

1) Reads messages from a remote IBM MQ(legacy system only works with IBM MQ)

2) Writes these messages to Kafka Topic

3) Reads these messages from the same Kafka Topic and calls a REST API.

4) There could be other consumers reading from this topic in future.

I came to know that Kafka has the new streams API which is supposed to be better than Kafka consumer in terms of speed/simplicity etc. Can someone please let me know if the streams API is a good fit for my use case and at what point in my process i can plug it ?

Solution

    1. Reads messages from a remote IBM MQ (legacy system only works with IBM MQ)

    2. Writes these messages to Kafka Topic

    I'd use Kafka Connect for (1) and (2). It is part of the Kafka project, and there are many free as well as commercial "connectors" available for hundreds of systems.

    1. Reads these messages from the same Kafka Topic and calls a REST API.

    You can use Kafka Streams as well as the lower-level Consumer API of Kafka, depending on what you prefer. I'd go with Kafka Streams as it is easier to use and far more powerful. (Both are part of the Kafka project.)

    1. There could be other consumers reading from this topic in future.

    This works out-of-the-box -- once data is stored in a Kafka topic according to step 2, many different applications and "consumers" can read this data independently.