Filebeat for shipping messages from a Kafka topic to Elasticsearch

Question

I'd like to have Filebeat set up the way that it would consume messages from Kafka topic and then send them to Elasticsearch. Is there a way, how to set it all up in docker-compose? I could register Logstash as a Kafka consumer so I assume that it should be also possible for Filebeat. In the log, I can see that it loaded the config file. I don't see it, however, in the list of Kafka's consumers. So Filebeat doesn't consume any messages, even though they are in Kafka. Can somebody point me in the right direction? I assume that it should be a really basic configuration. Yet, I'm missing something.

docker-compose.yml:

version: '3'
services:
  PostgreSQL:
    image: postgres:latest
    environment:
      - POSTGRES_DB=distillery
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=secret
    ports:
      - "5434:5432"

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "22181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS:         PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:     PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_RETENTION_MS: 10000
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000

  elasticsearch:
    image: elasticsearch:7.9.2
    depends_on:
      - kafka
    ports:
      - '9200:9200'
    environment:
      - discovery.type=single-node
    limits:
      memlock:
        soft: -1
        hard: -1

  filebeat:
    depends_on:
      - kafka
    image: docker.elastic.co/beats/filebeat:7.9.2
    container_name: filebeat
    volumes:
      - "./filebeat.yml"

The content of the filebeat.yml file:

  filebeat.inputs:
    - type: kafka
      hosts:
        - kafka:29092
      topics: ["progress-raspberry"]
      client_id: "filebeat"
      group_id: "filebeat"

 output.elasticsearch:
    hosts: ["localhost:9200"]`

Answer 1

Kafka advertised listener (the address clients connect to) is PLAINTEXT://kafka:9092. Therefore, that's the host/port you need Filebeat container to connect to.

If you ran Filebeat outside the container, then you'd use localhost:29092 for that advertised listener.

More details Connect to Kafka running in Docker