Search code examples
apache-kafkamicroservicesvert.x

can vert.x event bus replace the need for Kafka?


I am evaluating the vert.x framework to see if I can reduce the Kafka based communications between my microservices developed using spring boot.

The question is: Can I replace

  1. Kafka with vert.x event bus and
  2. spring boot microservices with vert.x based verticles

Solution

  • To answer quickly, I would say it depends on your needs.

    Yes, the eventbus can be a good way to handle natively communication between microservices verticles using an asynchronous and non-blocking paradigm.

    But in some cases you could need:

    • to handle some common enterprises patterns like replay mechanisms, persistence of messages, transactional reading
    • to be able to process some kind of messages in a chronological order
    • to handle communication between multiples kind of microservices that aren't all written with the same framework/toolkit or even programming language
    • to handle reliability, resilience and failure recovery when all your consumers/microservices/verticles are died
    • to handle dynamic horizontal scalability and monitoring of your consumers/microservices/verticles
    • to be able to work with a single cluster deployed in multi-datacenters and multi-regions

    In those cases I'd prefer to choose Apache Kafka over the native eventbus or an old fascioned JMS compliant system.

    It's not forbidden to use both eventbus and kafka in the same microservices architecture according to your real needs. For example, you could have one kafka consumers group reading a kafka topic to handle scaling, monitoring, failure recovery and reply mechanism and then handle communication between your sub-verticles through the eventbus.

    I'll clarify a little bit for the scalability and monitoring part and explain why I think it's more simple to handle that with Kafka over the native eventbus and cluster mode with vert.x : Kafka allow us to know in real time (through JMX metrics and the describe command):

    • the "lag" of a topic which corresponds to the number of unread messages
    • the number of consumers of each group that are listening a topic
    • the number of partitions of a topic affected of each consumers
    • i/o metrics

    So it's possible to use an ElasticStack or Prometheus+Grafana solution to monitor those metrics and use them to handle a dynamic scalability (when you know that there's a need to increase temporarily the number of consumers for example according to the lag metric and the number of partitions and the cpu/ram/swap metrics of your hosts).

    To answer the second question vert.x or SpringBoot my answer will be not very objective but I'd vote for vert.x for its performances on the JVM and especially for its simplicity. I'm a little tired of the Spring factory and its big layers of abstraction that hides a lot of issues under a mountain of annotations triggering a mountain of AOP.

    Moreover, In the Java world of microservices, there's other alternatives to SpringBoot like the different implementations of Microprofile (thorntail project for example).