Python rolling log to a variable

Question

I have an application that makes use of multi-threading and is run in the background on a server. In order to monitor the application without having to log on to the server, I decided to include Bottle in order to respond to a few HTTP endpoints and report status, perform remote shutdown, etc.

I also wanted to add a way to consult the logfile. I could log using the FileHandler and send the destination file when the URL is requested (e.g. /log).

However, I was wondering if it'd be possible to implement something like a RotatingFileHandler, but instead of logging to a file, logging to a variable (e.g. BytesIO). This way, I could limit the log to the most recent information, while at the same time being able to return it to the browser as text instead of as a separate file download.

The RotatingFileHandler requires a filename, so it's not an option to pass it a BytesIO stream. Logging to a variable itself is perfectly doable (e.g. Capturing Python Log Output In A Variable), but I'm a bit stumped on how to do the rolling part.

Any thoughts, hints, suggestions would be greatly appreciated.

Answer 1

Going further on Andrew Guy's suggestion, I subclassed logging.Handler and implemented a handler that uses collections.deque with a fixed length to keep a record of the log messages.

import logging
import collections


class TailLogHandler(logging.Handler):

    def __init__(self, log_queue):
        logging.Handler.__init__(self)
        self.log_queue = log_queue

    def emit(self, record):
        self.log_queue.append(self.format(record))


class TailLogger(object):

    def __init__(self, maxlen):
        self._log_queue = collections.deque(maxlen=maxlen)
        self._log_handler = TailLogHandler(self._log_queue)

    def contents(self):
        return '\n'.join(self._log_queue)

    @property
    def log_handler(self):
        return self._log_handler

Example usage:

import random

logger = logging.getLogger(__name__)

tail = TailLogger(10)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

log_handler = tail.log_handler
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)

levels = [logging.INFO, logging.ERROR, logging.WARN, logging.DEBUG, logging.CRITICAL]
logger.setLevel(logging.ERROR)

for i in range(500):
    logger.log(random.choice(levels), 'Message {}'.format(i))

print(tail.contents())

Output:

2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 471
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 472
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 473
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 474
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 477
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 481
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 483
2016-06-22 13:58:25,975 - __main__ - ERROR - Message 484
2016-06-22 13:58:25,975 - __main__ - CRITICAL - Message 485
2016-06-22 13:58:25,976 - __main__ - CRITICAL - Message 490