Python logging into several files across several modules

Question

I am dealing with the task how to setup logging in my project e.g. myApp. It consists of several modules and files and I want to create some basic pattern for logging. All logs should be located in /var/log/myApp. This directory should contain files e.g. user_errors.log and errors.log. Both files will have same format pattern with date, file, lineno etc. Now please consider following structure:

# a.py:
class A(object):
  def __init__(self):
    self.c = my_app.C()
  ...
  def some_method(self)
    raise MyUserError(...)

# b.py
class B(object):
  ...
  raise MyException(...)

# c.py
class C(object):
  ...
  raise MyException(...)


# d.py which has instances of A and B
class D(object):
  ...
  def some_method(self):
    ...
    except MyUserError as ue:
      # log into user_error.log
    except MyException as e:
      # log into error.log

The main approach is to log any exception which has been violated into different files according to the type of the exception. Also it is important to log the name and the line number where the exception was violated.I thought I would create some myLogger class which would somehow extend logging module, but I haven't found any good example how to do it. I have read the tutorial so I know how to format the output and so on. Thank you in advance for the help!

Answer 1

If you're catching all the errors at some centralized location (or a few centralized locations), you could just use two separate loggers with separate handlers.

log = logging.getLogger('errors')
userlog = logging.getLogger('user_errors')

error_handler = logging.FileHandler('/var/log/myApp/errors.log')
usererror_handler = logging.FileHandler('/var/log/myApp/user_errors.log')

log.addHandler(error_handler)
userlog.addHandler(usererror_handler)

def some_method(self):
    ...
    except MyUserError as ue:
        # log into user_error.log
        userlog.exception('User Error')
    except MyException as e:
        # log into error.log
        log.exception('Error')

If you have a lot of different exceptions and files to handle, you could create them like this

LOGGERS = {}

def create_loggers():
    logdir = '/var/log/appData'
    exceptions = [
        (MyError, 'errors.log'), 
        (MyUserError, 'user_errors.log'), 
        (OtherError, 'other.log')
    ]
    for exc, fn in exceptions:
        logger = logging.getLogger(exc.__name__)
        hdlr = logging.FileHandler(os.path.join(logdir, fn))
        logger.addHandler(hdlr)
        LOGGERS[exc] = logger

create_loggers()

Then in your exception hanlder code

except OtherException:
    LOGGERS[OtherException].exception('Error')