Search code examples
apache-kafkarhel

Kafka max throughput in messages per Second


What is the max throughput of Kafka in messages per second.

we wanted to check what is the theoretical max throughput that could be reached. in messages per second

we intend to use Apache Kafka version - 2.7.x and OS is RHEL 7.9

the physical machines with 125G memory and 32 Core per machine

links - https://aiven.io/blog/benchmarking-kafka-write-throughput-performance-2019-update


Solution

  • The max throughput is always going to be measured without network overhead (a local consumer/producer), and have no acks (message not durably or reliably written to disk), and no TLS encryption.

    Neither are useful tests or metrics to gather.

    Run your own benchmarking using performance test CLI commands included with Kafka, or external tools like Gatling/JMeter

    Kafka brokers are not RAM intensive, so each machine doesn't require 125G of memory. The disk speed and partitioning scheme will matter more, e.g. 7200 RPM w/ XFS formatted.

    https://docs.confluent.io/platform/current/kafka/deployment.html