Search code examples
restgoapache-kafkamicroservices

Is it possible to create a service that consume both message streaming (from Kafka) and also read REST API request, in Go?

I recently read an article Sharing Data Between Microservices on High Scale

There's this image that is about Asynchronous connection.
enter image description here

If I make no mistake, the Order Page service in this picture seems to be able to read messages streaming from Kafka, and also read REST API request from front-end app.

So, I wonder if it's possible to write a service that can do both :

  • Read messages streaming from Kafka
  • Read REST API request

In just 1 service, for instance in Go.

Also whether is it a good idea ?

Solution

  • Yes, you can definitely write a go service that consumes from Kafka and also exposes a REST endpoint. However, this is not a desirable design decision.

    The Kafka-based streaming system allows asynchronous communication between the producer and consumer. However, the REST API-based system does the synchronous communication between the caller and the callee.

    The availability guarantee between the sync and sync mode communication is different. The Kafka streaming-based system will have a different availability guarantee than the REST API. In the streaming system, we focus on the throughput of message processing, whereas in the sync communication end-to-end, latency is the main focus (you don't want to keep the caller waiting for a long time to get the result).

    If we put both these types of systems in a single service, it becomes difficult to gaze at the availability of the service. Also, the async part or the sync part of the service can play a part in a noisy neighbour for the other side, thus making the service unavailable.

    So in general, it is recommended to keep the REST-based API and Kafka streaming as two separate services.