Fluent-bit with java

Question

I'm trying to send the logs from a basic java maven project to fluent-bit configured on a remote machine. Fluent-bit would then write them to a file. This is my basic java configuration.

Java

private final static Logger logger = LoggerFactory.getLogger(App.class);

public static void main(String[] args) {
   for (int i = 0; ; i++) {
      logger.debug("Warn msg");
      try {
         Thread.sleep(5000);
       } catch (InterruptedException e) {
        // do nothing now
      }
   }
}

And the logback.xml

<appender name="fluentd" class="ch.qos.logback.more.appenders.DataFluentAppender">
   <remoteHost>xx.xxx.xxx.xxx</remoteHost>
   <port>7777</port>
   <encoder>
            <pattern>%message%n</pattern>
    </encoder>
</appender>

<root level="DEBUG">
    <appender-ref ref="fluentd" />
</root>

Fluent-bit configuration :

td-agent-bit.conf

[INPUT]
    Name          tcp
    Listen        xx.xxx.xxx.xxx
    Port          7777
    Parsers_File  /etc/td-agent-bit/parsers.conf
    Parser        custom_parser

[OUTPUT]
    Name          file
    Match         *
    Path          /home/td-agent-bit/output.txt

parsers.conf

[PARSER]
    Name    custom_parser
    Format  regex
    Regex   .*

I keep getting the following exception when the app runs

[2018/09/27 08:29:13] [trace] [in_tcp] read()=74 pre_len=370 now_len=444

[2018/09/27 08:29:13] [debug] [in_serial] invalid JSON message, skipping

But when I try testing the configuration via the command line it works

echo '{"key 1": 10, "key 2": "YYY"}' | nc xx.xxx.xxx.xxx 7777

I don't get any exception and the output file has all permissions. Also the remote machine is a photon-os based system.

Any ideas would be much appreciated.

Answer 1

So after some research and a ticket I opened here, I found out that I was using the wrong plugin.

All java configurations were correct. Just needed to make the following change to the td-agent-bit.conf

[INPUT]
    Name          forward
    Listen        xx.xxx.xxx.xxx
    Port          7777

We need to use the forward plugin instead of the tcp plugin. This plugin would listen to any incoming messages on the 7777 port and redirect it to the file.

Note that TCP Input plugin only accept JSON maps as records and not msgpack as forward protocol does.