Kafka streams in hexagonal architecture

Question

Im creating service in hexagonal architecture that consume data from topic. In the same projects I want to use kafka streams to merge few topics into single one and then consume that data from that topic created by kafka streams in my adapter. I have a problem where should I put kafka streams code in my hexagonal architecture? It don't seems that adapters are good place for that, either my domain. It's just topic data transformation that will be use later in those.

Answer 1

I once participated in a project that had the exact same use case (merging several topic to a single output topic that would be then consumed by a consumer of the service)

Kafka Streams in that regard is not really tied to your hexagonal architecture since it never actually interacts with your domain at all :)

Meaning the logic to build your Stream Topology makes sense only in the context of the Topology

In that particular project, we had the following layout :

  application
  |_ rest
  |_consumer // Module where your actual, runnable consumer resides, having 
  |          // dependecies on the domain and the infrastructure package
  |_ stream // Module harboring the topology itself, with no dependencies to 
  |          // the domain
  domain
  |_core // Domain implementation
  |_contract // Domain contract (DTO and ports definition)
  infrastructure
   

We deployed the stream application in an isolated container, and its module had no interaction/dependencies towards the domain

So, to summarize : you can treat Kafka Stream as an independent "application" module (that is, module whose build will produce an executable artifact) if you wish to colocate it in the same project (meaning in the same codebase).

If you do not have the need or the will to colocate it in the same codebase, it could reside in a repository of its own, because it will never be tied to your domain logic.

You can see it as a technical component making a sort of Kafka Topics arithmetic of sort (topic A + topic B = topic C) that has in the end no ties nor impact to your domain.

The only dependency between your stream and the rest of your service would be the topology output topic contract, that would be a dependency of your consumer :)

Hope that helps, feel free to reach out if you have any questions/need any precisions :D