Celeryd - send an email on log level error and above

Question

We have django configured to send us emails on any error or above which is triggered. This is done using the standard LOGGING configuration in Django. I want this same behavior in celery. I have it working to send me emails on Exceptions ([CELERY_SEND_TASK_ERROR_EMAILS][2]), but I want an email on any defined level - coincidentally error and above.

For example in any django files we can do this.

log = logging.getLogger(__name__)
log.error("Oh No!")

And voilà it will send us an email assuming the following is setup in settings.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'standard': {
            'format': "[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
            'datefmt': "%d/%b/%Y %H:%M:%S"
        },
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True,
        },
    },
    'loggers': {
        '': {
            'handlers': ['logfile', 'mail_admins'],
            'level': os.environ.get('DEBUG_LEVEL', 'ERROR'),
        },
    }
}

And for clarity I am calling celeryd like so.

../bin/python manage.py celeryd --settings=DJANGO_SETTINGS_MODULE \
    --broker=amqp://RABBITMQ_USER:RABBITMQ_PASSWORD@localhost:5672/axis \
    --beat --events --pidfile=/home/var/celeryd.pid \
    --logfile=/home/logs/celeryd.log \
    --loglevel=WARNING > /dev/null 2>&1

And the necessary celery related settings.

CELERY_SEND_EVENTS = True
CELERY_TRACK_STARTED = True
CELERYD_CONCURRENCY = 2
CELERYD_TASK_TIME_LIMIT = 60 * 60 * 2    # Kill anything longer than 2 hours
CELERY_TASK_RESULT_EXPIRES = 60 * 60 * 2
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_SEND_TASK_ERROR_EMAILS = True
CELERY_MAX_PREPARING = 600 

And finally a basic test case which I feel should send two emails. One for the error and the other for the exception..

from django.contrib.auth.models import User

import celery
from celery.schedules import crontab
from celery.utils.serialization import UnpickleableExceptionWrapper
from celery.utils.log import get_task_logger

log = get_task_logger(__name__)

@periodic_task(run_every=datetime.timedelta(minutes=5))
def noise_test(**kwargs):
    log.info("Info NOISE TEST")
    log.warning("Warning NOISE TEST")
    log.error("Error NOISE TEST")

    try:
        User.objects.get(id=9999999)
    except Exception as err:
        from celery import current_app
        err.args = list(err.args) + ["Crap"]
        raise UnpickleableExceptionWrapper(
            err.__class__.__module__, err.__class__.__name__, err.args)

Answer 1

I'm looking for ways to improve this and gain a deeper level of understanding (@asksol) but here is what I came up with.

This snippet can be placed in any *.tasks.py folder as it's global and will get picked up.

from celery.signals import after_setup_task_logger
def setup_logging(**kwargs):
    """
      Handler names is a list of handlers from your settings.py you want to 
      attach to this
    """

    handler_names = ['mail_admins']

    import logging.config
    from django.conf import settings
    logging.config.dictConfig(settings.LOGGING)

    logger = kwargs.get('logger')

    handlers = [x for x in logging.root.handlers if x.name in handler_names]
    for handler in handlers:
        logger.addHandler(handler)
        logger.setLevel(handler.level)
        logger.propagate = False

after_setup_task_logger.connect(setup_logging)

And let the email flow!!

Couple Notes:

• mail_admins corresponds to the settings.LOGGING.handlers which has the magic django.utils.log.AdminEmailHandler class bound to it.
• Make sure that you disable DEBUG as email won't fly if DEBUG = True
• Make sure that you set CELERY_SEND_TASK_ERROR_EMAILS = not DEBUG
• Make sure that your caller sets the logging level to something <= the level you want to capture.
• What is REALLY cool is then in the example above you want to accurately record the exception do this.

Try this..

@celery.task()
def dummy_test(**kwargs):

    log = dummy_test.get_logger()
    if settings.DEBUG:
        log.setLevel(logging.DEBUG)

    log.debug("Debug DUMMY TEST")
    log.info("Info DUMMY TEST")
    log.warning("Warning DUMMY TEST")
    log.error("Error DUMMY TEST")
    log.critical("Critical DUMMY TEST")

    try:
        User.objects.get(id=9999999)
    except Exception as err:
        log.exception(err)
        raise

Now the exception will get sent in a nice email. And each message gets printed..