Can't connect to kafka docker container from another docker container: Kafka:3.7?

Question

This question is quite similar to this one here, yet I still can't figure out why the following docker-compose.yml, copied mainly from official site, not working when I am trying to connect to it from another docker container.

My docker-compose.yml, Kafka server starts successfully

version: '3'

services:
  kafkabroker:
    image: apache/kafka:latest
    hostname: kafkabroker
    container_name: kafkabroker
    ports:
      - '9092:9092'
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://kafkabroker:9092,PLAINTEXT://kafkabroker:19092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafkabroker:29093'
      KAFKA_LISTENERS: 'CONTROLLER://kafkabroker:29093,PLAINTEXT_HOST://kafkabroker:9092,PLAINTEXT://kafkabroker:19092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'

My simple python code to connect to above Kafka container:

docker run -it --rm --mount type=bind,src=/Users/myuser/python/,target=/app -w /app python-kafka python /app/kproducer.py

And the python code:

from confluent_kafka.admin import AdminClient

adminClient = AdminClient({
    "bootstrap.servers": "kafkabroker:9092",
})

print(adminClient)

The error keeps saying: Name or service not known:

%3|1712683405.988|FAIL|rdkafka#producer-1| [thrd:kafkabroker:9092/bootstrap]: kafkabroker:9092/bootstrap: Failed to resolve 'kafkabroker:9092': Name or service not known (after 2ms in state CONNECT)

I know that it would be a very simple question, yet if anyone can help me on this, I would be really appreciated.

Thank you very much in advance for all your help.

Answer 1

You can create a docker network and run the python application also in the docker-compose.yaml.

If not you can use below approach

Add a network to docker-compose.yaml

version: '3'

networks:
  sample_network:
    driver: bridge

services:
  kafkabroker:
    image: apache/kafka:latest
    hostname: kafkabroker
    container_name: kafkabroker
    networks:
      - sample_network
    ports:
      - '9092:9092'
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT_HOST://kafkabroker:9092,PLAINTEXT://kafkabroker:19092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafkabroker:29093'
      KAFKA_LISTENERS: 'CONTROLLER://kafkabroker:29093,PLAINTEXT_HOST://kafkabroker:9092,PLAINTEXT://kafkabroker:19092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'

Finally when you are running your python application, attach the above "sample_network" to it

docker run -it --rm --network sample_network --mount type=bind,src=/Users/myuser/python/,target=/app -w /app python-kafka python /app/kproducer.py