tomcat logging rsyslog nxlog papertrail-app

Multiline Log issue with rsyslog

I have an application server (Ubuntu 14.04) which has tomcat server running on top of it. This application server uses "rsyslog" which is configured to send the logs to a NXlog server (on Ubuntu 14.04).

Rsyslog sends all its logs, including the tomcat errors, exceptions & stack traces to syslog server, but there is a problem with multiline logs. When the log messages are stored in files or forwarded over the network without any encapsulation, the newline character present in messages spanning multiple lines confuse simple linebased parsers which treat every line as a separate event; & hence my exception logs get broken in new lines.

My rsyslog version is : 7.4.4

The rsyslog.conf file looks like this :

#################
#### MODULES ####
#################
$EscapeControlCharactersOnReceive off
$LocalHostName nishant-app

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)i
#$ModLoad immark  # provides --MARK-- message capability
$ModLoad imfile
$ModLoad omrelp
#$ModLoad omhdfs

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

module(load="imfile" PollingInterval="10")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++#

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf           ## This includes all the conf files which tells rsyslog which logs need to be sent

So basically I need to send the tomcat stack-traces & exception such that exceptions don't get scattered in multiple lines.

I am looking to solve this at rsyslog end but also confused that weather this can be solved at Nxlog server side also ?

Solution

I think the answer depends on how your logs end up in rsyslog. If there's an appender than sends stuff to the syslog socket, it's up to it. As far as I know, you can send multiline logs there, but if the appender breaks them before getting to rsyslog, there's not much you can do there. The same applies to UDP forwarding, each packet is a log, so rsyslog just takes it as it gets it.

If it sends via TCP, the default delimiter for messages is newline. Though rsyslog supports octet-delimited framing, this is again something to handle on the sender side.

If you're tailing files however, this is where you can do something. Though you'll probably need the latest version of rsyslog (there are Ubuntu packages here). With it, you'll get two important features for this usecase:

inotify mode (which is used by default). Much nicer than polling mode in terms of performance and playing nicely with log rotation
startmsg.regex (which allows you to specify a regex to figure out which line should belong to the current event and which should start a new one)

The point is, with multiline logs you (or rather, rsyslog) have to figure out another way of delimiting messages. With imfile, even in 7.4.4, there's the option of using ReadMode, which defaults to 0 (newline is a delimiter), but you can set it to 2 (if line begins with space/tab it belongs to the previous one).

You can find all the options around imfile here: http://www.rsyslog.com/doc/master/configuration/modules/imfile.html