Search code examples
sslapache-kafkaload-balancing

Using load balancer to terminate Kafka/SSL connections

We have two Kafka nodes and for reasons outside the scope of this question, would like to set up a load balancer to terminate SSL with producers (clients). The SSL cert hosted by the load balancer will be signed by trusted/root CA that clients should natively trust.

So the connection would look like:

Kafka Producers <---SSL/Port 9094--> Load Balancer <---TCP/Port 9092 ---> Kafka Node

Is this possible to do, or does Kafka somehow require SSL to be setup directly on the Kafka servers themselves?

Thanks!

Solution

  • It turns out Kafka has its own internal load balancing protocol, and adding any 3rd party balancer in between Kafka clients and their brokers interferes with this protocol and causes very strange behavior.

    Don't do this.

    My motivation for wanting to do this was because I had failed to get SSL properly working between clients and brokers, and I thought terminating SSL at a load balancer, and then allowing plaintext communication between balancer and brokers would solve my issue. It did not. Don't do this.

    Either:

    1. Figure out how to get SSL working between clients and brokers (the official Kafka user mailing list was excellent at helping me achieve this); or
    2. Don't require/support an SSL connection; or
    3. Put a web service in front of your Kafka