Kafka Producer, Consumer, Broker in same host?

Question

Are there any downsides to running the same producer and consumer code for all nodes in the cluster? If there are 8 nodes in the cluster (8 consumer, 8 kafka broker, and 8 producers), would 8 producers be running at the same time in the cluster then? Is there a way to modify cluster so that only one producer runs at a time?

Answer 1

Are there any downsides to running the same producer and consumer code for all nodes in the cluster?

The primary downsides here would be scalability and memory usage.

Producers and Consumers are not required to run on Brokers. Producers should be deployed where data is being generated (or running as separate hosts, like Kafka Connect workers).

Consumers should be scaled out independently based on the throughput and ordering guarantees that you need in your downstream systems.

There is nothing that says 8 brokers requires 8 producers and 8 consumers; partitions are what matters more

• If you have N partitions in a topic, you can only scale to N active consumers anyway, and infinitely many producers
• 8 brokers can hold lots of partitions for any given topic

Running a single producer is an implementation of your own code. The broker cannot force it.