Search code examples
pythontcpsyslogfluentd

Python's SyslogHandler and TCP

I'm trying to understand why the SyslogHandler class from Python's logging framework (logging.handlers) does not implement any of the framing mechanism described by RFC 6587:

  1. Octet Counting: it "prepends" the message length to the syslog frame:

  2. Non-Transparent-Framing: a trailer character to separate messages. This is what most of the servers understand.

This "problem" can be easily solved by adding a LF character to the end of the messages, however I would expect that the SyslogHandler would take care of this by default:

sHandler = logging.handlers.SysLogHandler(address=(address[0], address[1]), socktype = socket.SOCK_STREAM)
sHandler.setFormatter(logging.Formatter(fmt=MSG_SYSLOG_FORMAT, datefmt=DATE_FMT))
self.addHandler(sHandler)

This does not work neither with Fluentd, nor with rsyslog. As I said, I've temporarily added this to line 855 of handlers.py (just to test):

msg = prio + msg + '\n'

And now is working.

My questions:

  1. Should the Python SyslogHandler class offer the possibility to set on/off the octet counting or trailer character. Currently it does nothing...
  2. It is the job of the programmer to know how the server works and override the Handler to address the message framing?

For now, what I'm doing now is to override emit() method, sub-classing SyslogHandler.

Solution

  • Syslog support in logging predates the RFC, and before that RFC, there was little in the way of standards.

    To be precise: the SysLogHandler handler was part of logging when first added to the Python standard library in 2002 and has remained largely the same since (TCP support was added in 2009, and RFC5424 support was improved in 2011); the original code was based on this syslog module from 1997.

    From other bug reports it is clear the maintainers want to keep the broadest backwards compatibility in code here, so if you need specific functionality from a newer RFC, you have two options:

    • Extend the class and implement that functionality yourself
    • Submit feature requests and / or patches to improve the functionality in the logging module; take into account the backwards-compatibility requirements.