We have configured our application to write some specific log messages to System's Syslog file using the Syslog appender of Log4j2. No issue in writing the Syslog to the file. But when the syslog service is restarted, the first log message is not written to the syslog. The subsequent messages are written.
Enabled debug logs of Log4j, no exception is seen while writing 1st message to syslog after the restart. But for the subsequent request, the following messages were captured in the Log4j2 log.
2022-01-27 18:07:40,120 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Reconnecting localhost/127.0.0.1:514
2022-01-27 18:07:40,121 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Creating socket localhost/127.0.0.1:514
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Closing SocketOutputStream java.net.SocketOutputStream@1a769d7
2022-01-27 18:07:40,122 ajp-nio-0.0.0.0-8009-exec-3 DEBUG Connection to localhost:514 reestablished: Socket[addr=localhost/127.0.0.1,port=514,localport=57852]
I took threaddump and checked whether the Reconnector thread is running but no such exists in the threaddump. I am clueless here, any help on finding the reason for missing the message would be helpful.
Environment details: CentOS 7.9 + RSyslog Service, Application deployed in Tomcat and running on Java 11, Log4j2 version is 2.17.1
This is due to the way plain text TCP syslog works. Check out this post for further information.
This "bug" exists, since version 8.1901 and newer.
The only way you can fix this - as far as i know - is to send the messages over the RELP protocol. See omrelp module.