Search code examples
fluent-bit

Fluent Bit not producing output

I'm running Fluent Bit on Mac M2. I'm trying to process KubeApi server log file (10 records). When I run FluentBit - I've got empty output. If I replace my INPUT with dummy input - I've got fine output.

I have following Fluent Bit config:

[SERVICE]
    Daemon Off
    Flush 1

[INPUT]
    Name tail
    Path /Users/me/temp/fluentbit/source/*.log

[OUTPUT]
    Name file
    Match *
    File out.log
    Path /Users/me/temp/fluentbit/target

Logs from run:

[2023/07/20 21:56:11] [ info] [fluent bit] version=2.1.7, commit=, pid=494
[2023/07/20 21:56:11] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/20 21:56:11] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/20 21:56:11] [ info] [cmetrics] version=0.6.3
[2023/07/20 21:56:11] [ info] [ctraces ] version=0.3.1
[2023/07/20 21:56:11] [ info] [input:tail:tail.0] initializing
[2023/07/20 21:56:11] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/07/20 21:56:11] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/07/20 21:56:11] [ info] [input:tail:tail.0] multiline core started
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] scanning path /Users/me/temp/fluentbit/source/*.log
[2023/07/20 21:56:11] [ info] [output:file:file.0] worker #0 started
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] inode=81899958 with offset=2843 appended as /Users/me/temp/fluentbit/source/kube-apiserver-kube-master_kube-system_kube-apiserver-67b2090d83e1f84c52f2c0fd516828d723fb26d6272b039739cdcc0c39604fe4.log
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] scan_glob add(): /Users/me/temp/fluentbit/source/kube-apiserver-kube-master_kube-system_kube-apiserver-67b2090d83e1f84c52f2c0fd516828d723fb26d6272b039739cdcc0c39604fe4.log, inode 81899958
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] 1 new files found on path '/Users/me/temp/fluentbit/source/*.log'
[2023/07/20 21:56:11] [debug] [file:file.0] created event channels: read=29 write=30
[2023/07/20 21:56:11] [ info] [sp] stream processor started
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] inode=81899958 file=/Users/me/temp/fluentbit/source/kube-apiserver-kube-master_kube-system_kube-apiserver-67b2090d83e1f84c52f2c0fd516828d723fb26d6272b039739cdcc0c39604fe4.log promote to TAIL_EVENT
[2023/07/20 21:56:11] [debug] [input:tail:tail.0] [static files] processed 0b, done
^C[2023/07/20 21:56:33] [engine] caught signal (SIGINT)
[2023/07/20 21:56:33] [ info] [input] pausing tail.0
[2023/07/20 21:56:33] [debug] [input:tail:tail.0] inode=81899958 removing file name /Users/me/temp/fluentbit/source/kube-apiserver-kube-master_kube-system_kube-apiserver-67b2090d83e1f84c52f2c0fd516828d723fb26d6272b039739cdcc0c39604fe4.log
[2023/07/20 21:56:33] [ info] [output:file:file.0] thread worker #0 stopping...
[2023/07/20 21:56:33] [ info] [output:file:file.0] thread worker #0 stopped

I've tried a lot of variation of input with different parsers, but without result. Any ideas why output is empty?

Solution

  • I found out how to produce output.

    You need to start Fluent Bit and after apply new line to your LOG source file. So Fluent Bit starts to flush.

    For example:

    echo "$(date -u +"%Y-%m-%dT%H:%M:%S.%N%Z") stderr F [$(date +"%Y/%m/%d %H:%M:%S")] [error] [output:gelf:gelf.0] error encoding to GELF" >> my-log-file.log