How to output and log messages at the same time in python

Question

I want something like what the tee command does in Linux based systems, but in pure python code.

I have tried using this snippet:

import sys

# Define the file name for logging
log_file_name = 'output.log'

# Open the file in append mode
log_file = open(log_file_name, 'a')

# Function to log output to both console and file
def log(message):
    # Write to console
    print(message)
    # Write to file
    log_file.write(message + '\n')
    # Flush the buffer to ensure the message is written immediately
    log_file.flush()

# Redirect stdout and stderr to the log function
sys.stdout.write = log
sys.stderr.write = log


print("Hello world!")

However I get this error:

object address  : 0x7f88e6b1c1c0
object refcount : 2
object type     : 0x9e14c0
object type name: RecursionError
object repr     : RecursionError('maximum recursion depth exceeded')
lost sys.stderr

and whenever I comment out the print statement in the log function it logs the message into the log file but does not output the message into the console. Why is this happening and how can I fix it?

Answer 1

What goes wrong

print in python writes the given message to stdout. The reason you're getting this error, is that you've rebound sys.stdout.write to your log function, but inside this log function, there is also a call to print. The line print(message) now again calls stdout.write, which is the log function. This happens recursively until you hit the max recursion depth, which is 1000 by default in python.

Solution in python

I don't think you should want to do what you're currently doing. By overwriting sys.stdout.write, you're hacking your own logging into python, which may well break any packages you're using, or at the very least it will greatly confuse anyone you're working with.

It would probably be simplest to just write something like the following, and replace print statements with output statements in your code.

def output(message):
    # To console
    print(message)

    # To file
    with open(log_file_name, 'a') as file:
        file.write(message + '\n')

Note that you were also not closing the file after writing to it, potentially risking data corruption. See for instance this explanation for why with statements are your greatest friend!

But if possible, using tee probably is the simplest solution.

EDIT:

Note on the logging module

Actually, such functionality is also possible using the logging module from the standard library, for a more production-ready solution. The following configuration writes to both the console and a file every time the test logger gets used:

LOGGING = {
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
        "file": {
            "class": "logging.FileHandler",
            "filename": "debug.log",
        },
    },
    "loggers": {
        "test": {
            "handlers": ["console", "file"],
        },
    },
}