Kafka with Docker Swarm - How to execute Kafka commands

Question

I am new to Kafka and I am a little confused.

So far I have created two docker machines, a manager and a worker with these commands:

docker-machine create manager
docker-machine create worker1

I have add these two nodes inside a docker swarm.

docker@manager:~$ docker node ls                                                                                                                                                                             
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
6bmovp3hr0j2w5irmexvvjgzq *   manager             Ready               Active              Leader              19.03.5
mtgbd9bg8d6q0lk9ycw10bxos     worker1             Ready               Active                                  19.03.5

I want to create a kafka cluster inside docker swarm. My docker-compose.yml looks like this

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "hostname | awk -F'-' '{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

where hostname | awk -F'-' '{print $$2}' command returns manager of worker.

Then I deploy with the command docker stack deploy -c docker-compose.yml kafka which returns

Creating network kafka_default
Creating service kafka_zookeeper
Creating service kafka_kafka

I checked that th service is running with docker service ls

ID                  NAME                MODE                REPLICAS            IMAGE                           PORTS
ttprq3ak98om        kafka_kafka         replicated          0/1                 wurstmeister/kafka:latest       
vxdhkaonrbpe        kafka_zookeeper     replicated          0/1                 wurstmeister/zookeeper:latest   *:2181->2181/tcp

and then I scale the service with docker service scale kafka_kafka=2. When I execute docker service ps kafka_kafka I get

ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ydn64mwvlasr        kafka_kafka.1       wurstmeister/kafka:latest   worker1             Running             Running 27 minutes ago                       *:9094->9094/tcp
shkpd2jv6s29        kafka_kafka.2       wurstmeister/kafka:latest   manager             Running             Running 27 minutes ago                       *:9094->9094/tcp

So far I have one ZooKeeper and two Kafka brokers. One broker on manager and one on worker. This means that the broker on worker replicates the broker on manager?

Also I wonder how to run Kafka commands in this step. Suppose that I want to create a topic. How can I achieve this? I have to download Kafka binary from here: https://kafka.apache.org/downloads? And if the answer is yes, I have to strat the zookeeper server? Is it not running already?

Answer 1

to run kafka commands inside the container you need to first get the kafka container id

$ docker ps -a  
  CONTAINER ID                 IMAGE  
   1239075412         wurstmeister/kafka:latest  

now take this container id(will be different in your case) of kafka

$ docker exec -it 1239075412 sh  

logs are present in the following folder

# cd kafka/kafka-logs-1239075412  

kafka commands will be present in this container type kafka- and then hit tab twice
you will get all the kafka commands to create topics etc..