How to connect with Kafka producer into docker container running in another container on same VM?

Question

I have a simple Scala application producing messages into Kafka. Both Kafka and Scala application are running in separate containers on the same VM.

@main def kafkaTest: Unit =
  // Kafka producer initialisation
  val kafkaProps = new Properties()

  kafkaProps.put("bootstrap.servers", "localhost:9092")
  kafkaProps.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
  kafkaProps.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
  kafkaProps.put("linger.ms", "0")

  val producer =  new KafkaProducer[String, String](kafkaProps)

  producer.send(new ProducerRecord[String, String]("test-topic", "Test message"))
  producer.close()

If I build and run the Scala application directly on the VM (not in a container) the code works as expected. However, if I run the code in a Docker container, then the producer does not seem to be able to connect to Kafka (no messages are written to Kafka).

I also tried running the docker container with host network, but still facing the same issue: docker run <image_name> --network host

Thus, I suspect this is a Docker/networking issue as the code works as expected if ran directly on the VM and not in a container.

Docker compose file used to set up Kafka:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost

So my question is: how can I connect from the Scala application running in a container to Kafka running in another container (on the same VM)?

Answer 1

The error was in the command used to launch Scala container. The --network option needs to come before the image name:

docker run --network=host <image_image>

instead of

docker run <image_name> --network=host