How to remove BOM (byte order mark) from rsyslog messages

Question

In my current rsyslog messages, there are a lot BOMs showing up like <feff> in the log messages. Is there any way I can get rid of these BOMs in the log messages? I'm using rsyslog 8.10. Any suggestions? Thanks in advance.

Answer 1

I think this might actually be caused by a bug in older versions of Python. Do you know if your Python version on the system is up to date? If not, you might try updating it to whatever the latest Python 2.7 is that you can get for the system.

Related bug reports:

• Prepending BOM to UTF8 message before sending to rsyslog causes bad characters in log http://bugzilla.adiscon.com/show_bug.cgi?id=346
• UTF8 BOM incorrectly prepended syslog messages when using rsysolog http://bugs.python.org/issue15462
• SysLogHandler sends invalid messages when using unicode
  http://bugs.python.org/issue14452

Short of updating to a new Python 2.7 version, tt seems the only way to work around this is for an older Python 2.7 install is to apply the following patch to the /usr/lib/python2.7/logging/handlers.py file on your system.

@@ -797,9 +797,10 @@
                                             self.mapPriority(record.levelname))
         # Message is a string. Convert to bytes as required by RFC 5424
         if type(msg) is unicode:
            msg = msg.encode('utf-8')
- if codecs:
- msg = codecs.BOM_UTF8 + msg
+ #if codecs:
+ # msg = codecs.BOM_UTF8 + msg
         msg = prio + msg
         try:
             if self.unixsocket:

For a Python 2.6 system, I think you might instead need the following patch to the /usr/lib/python2.6/logging/handlers.py file on the system.

@@ -821,8 +821,6 @@ class SysLogHandler(logging.Handler):
         # Message is a string. Convert to bytes as required by RFC 5424
         if type(msg) is unicode:
             msg = msg.encode('utf-8')
-            if codecs:
-                msg = codecs.BOM_UTF8 + msg
         msg = prio + msg
         try:
             if self.unixsocket: