Looking for pros and cons of using Apache Kafka over RabbitMQ. Also to decide if I should move my existing infrastructure over to Kafka.
very different, some you might consider to begin with - a) rabbit mq is queue FIFO. kafka is a log, your writes are appended to the tail, but you read from where you want.
b) Kafka is truly distributed - data is sharded , replicated, durability guarantees can be tuned, availability can be tuned. rabbitmq has limited support for the above.
c) Kafka also comes OOB with consumer frameworks which allow distributed processing of the log reliably. Kafka streams also has stream processing semantics built into it. rabbitmq the consumer is just FIFO based, reading from the HEAD and processing 1 by 1.
d) Kafka is extensible in the consumer model, allows you to build exactly once, atmost once , atleast once.