Search code examples
pythonloggingstdoutstderrobservability

Python default and multi level logging

From what I read and understood, Python logging module by default logs to stderr.

If I run this python code:

import logging


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

as

python main.py 1> stdout.txt 2> stderr.txt

I get my logs in stderr.txt and nothing in stdout.txt - my logs are redirected to stderr.

This default behaviour is problematic when logs are streamed to logging aggregation services such as datadog or papertrail. Since its streamed to stderr, the logs are marked as errors when in reality they are not.

So I tried to create multiple log handlers as follows:

import logging
import sys


stdoutHandler = logging.StreamHandler(stream=sys.stdout)
stderrHandler = logging.StreamHandler(stream=sys.stderr)

logging.basicConfig(level=logging.DEBUG, handlers=[stdoutHandler, stderrHandler])

stdoutHandler.setLevel(logging.DEBUG)
stderrHandler.setLevel(logging.ERROR)


logging.info('test')
logging.warning('test')
logging.error('test')
logging.debug('test')

When I run this code, I get errors in sterr.txt but also all the logs in stdout.txt - I ended up having log duplication error logs appear in both the stderr and stdout streams.

Is there a better way to handle the differentiation of error logs from the rest in Python?

I tried loguru package as well, also no luck in stream separation... Thanks in advance

Solution

  • You can probably benefit from the approach described here in the official documentation.