Apache Kafka local development

Question

I am wondering - how do you develop applications that produce/consume messages with Apache Kafka? I have been looking into the Strimzi operator for Kubernetes and actually deployed it on the cloud provider. I have tested it with the provided commands - running two pods (1 producer, 1 consumer) works fine and the messages get delivered.

However, I want to consume Apache Kafka with the applications I am developing. I have been trying to port-forward into the Kafka service, however, was not successful - requests doesn't seem to pass. I have been trying to setup a minikube cluster with the same operator - I am not able to connect to the cluster through any applications, just the testing setup with consumer/producer pods.

So, eventually, my question is - what is your setup in developing applications with Apache Kafka on Kubernetes? How do you test your code, write the code, check if business logic works correctly with Kafka?

Answer 1

You cannot use port forwarding with Kafka easily (this blog post series explains why: https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/). I think you have several options to use Strimzi for lcoal development:

• Use an external type listener such as node ports, loadbalancers or Ingress. This is not always simple with local Kubernetes clusters such as Minikube or Kind as they do not always support these things (especially on Windows or MacOS where they often run as a container inside a VM and node ports and loadbalancers often don't work) and Ingress is not always simple to use.
• You can run Kubernetes with some public cloud that has better support for things such as laod balancers or node ports that make it easier to expose the Kafka cluster outside of Kubernetes.
• You can try (my own) utility Kekspose that uses port-forwarding coupled with the Kroxylicious Kafka proxy to mock the advertised listeners. This does not require the external listener and should work fine with some Kubernetes cluster in the cloud as well as with Minikube.

You should also consider what does running Kafka on Kubernetes for local development really give you. If you just want to send / receive messages from the app you develop, the easiest way to run Kafka might be actually to just download the Kafka ZIP and run it locally. But this might not be that useful if you also need to run some other Kubernetes applications you integrate with through Kafka. So it depends how suitable it is for you.