Docker container not sending logs to Fluentbit

Question

fluent-bit.conf

[INPUT]
    Name http
    Listen 0.0.0.0
    port 24224

[OUTPUT]
    Name  stdout
    Match **

Fluentbit Dockerfile

FROM fluent/fluent-bit:2.1.8
ADD fluent-bit.conf /fluent-bit/etc/

docker-compose.yaml

services:
  fluentbit:
    image: 'shubham01/fluentbit:latest'
    container_name: fluentbit
    hostname: fluentbit
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    volumes:
      - ./fluentbit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./log/:/etc/data
  app:
    container_name: app
    hostname: app
    image: app
    stdin_open: true
    tty: true
    build:
      context: .
      dockerfile: Dockerfile
      args:
        END: 'TEST'
    volumes:
      - "/tmp:/tmp"

    ports:
      - '8080:8080'

    restart: 'on-failure'

    depends_on:
      - fluentbit

    logging:
      driver: fluentd
      options:
        fluentd-address: "fluentbit:24224"
        fluentd-async-connect: "true"
        tag: 'containerssh.{{.ID}}'

Fluent bit container logs

Fluent Bit v2.1.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2024/01/14 09:02:55] [ info] [fluent bit] version=2.1.8, commit=1d83649441, pid=1
[2024/01/14 09:02:55] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/14 09:02:55] [ info] [cmetrics] version=0.6.3
[2024/01/14 09:02:55] [ info] [ctraces ] version=0.3.1
[2024/01/14 09:02:55] [ info] [input:http:http.0] initializing
[2024/01/14 09:02:55] [ info] [input:http:http.0] storage_strategy='memory' (memory only)
[2024/01/14 09:02:55] [ info] [output:stdout:stdout.0] worker #0 started

I'm trying to get the logs from docker containers to a central place so that they can be forwarded anywhere.

Logs are showing in container logs of service app but not showing on stdout of Fluentbit.

Dockerd is executing on Ubuntu.

1. Is it right to do what I'm doing ?
2. Is this right way to do what I'm doing ?
3. Is there a way fix this ?
4. Is this a problem with different network being used ?
5. Is http input right way for fluentbit to receive logs from other containers ?

Answer 1

In the configuration there is two issue

• invalid host to fluentbit (should be localhost)
• incorrect INPUT for you data (name should be forward)

Another thing to mention is using fluentd-async-connect: "true" make debugging it hard, because when host is not available you will wan't get any error

Issue #1
To understand why in this case localhost is correct you have to think from perspective of Docker Daemon not container app, app is only generating logs but sending it is done by Docker Daemon. Looking into docker docs we can find:
To use this logging driver, start the fluentd daemon on a host. We recommend that you use the Fluentd
You are starting it as a container, but as you have port exported - "24224:24224" is fully available by the host, but you have to use localhost instead of container name.

Issue #2
You are using INPUT with name http that is not valid reader for data send by Docker Daemon, diving more into Docker documentation you can find example Test container loggers that shows @type forward(it's like name in you config) all this happen because docker already has implemented fluentd logging protocol.

When you have INPUT name http fluentd is expecting raw HTTP request that you can be send using curl(based on example from fluent bit):

curl -d '{"key1":"value1","key2":"value2"}' -XPOST -H "content-type: application/json" http://localhost:24224/app.log

Bonus
When you have to troubleshoot docker logging to fluent bit it's better to disable fluentd-async-connect: "true" just remove or set to false because it will help you debug connection between Docker Daemon and flient bit

Changed example that I used to verify it (I changed your app to nginx)

services:
  fluentbit:
    image: 'shubham01/fluentbit:latest'
    container_name: fluentbit
    hostname: fluentbit
    ports:
      - "24224:24224"
    volumes:
      - ./fluentbit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./log/:/etc/data
  app:
    container_name: nginx
    hostname: nginx
    image: nginx
    stdin_open: true
    tty: true
    volumes:
      - "/tmp:/tmp"

    ports:
      - '8080:80'

    restart: 'on-failure'

    depends_on:
      - fluentbit

    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        tag: 'containerssh.{{.ID}}'

fluent-bit.conf after fixes

[INPUT]
    name forward
    listen 0.0.0.0
    port 24224

[OUTPUT]
    name  stdout
    match *