How to log in the celery log, outside celery tasks (inside a django view)?

Question

Suppose an endpoint in a django project

def my_view(request, my_id):
    if is_request_valid(request):
        logger = logging.getLogger('celery.task')
        logger.info('Starting a task for a thing with ID %i', my_id)
        my_cool_task.apply_async()

Now, according to the celery docs this may log, since

A special logger is available named “celery.task”, you can inherit from this logger to automatically get the task name and unique id as part of the logs.

But the "celery.task" logger works only inside a task. So in the view this log would go to nowhere.

Also, I did play with setting some logging variable inside settings.py, and then reassigning the celery's logging. Did play with the CELERY_WORKER_HIJACK_ROOT_LOGGER, but even when I use the root logger, the logs do not appear in the console. Nor using getLogger("celery") does not help.

This seems like a simple thing to do, but I'm struggling all day to do this, please help.

UPD: I've seen a post on reddit with the same problem, but there are no answers

Answer 1

So, as the question stands, it is impossible to hack into the celery's logger.

However, there are a few workarounds:

1. Make a task for logging, e.g.

from celery.utils.log import get_task_logger

@shared_task
def log_stuff(level, message, *args, **kwargs):
    get_task_logger().log(level, message, *args, **kwargs)

However, this is dirty for many reasons: mainly because your log will get executed when the worker will get to it, and so this just becomes a mess.

2. Log into the same file. Go into your_projects_name/settings.py, define LOGGING variable, for example:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False, 
    'formatters': {
        'simple': {
            'format': '[%(levelname)s] %(message)s',
        },
    },
    'handlers': {
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        'file': {
            'level': 'INFO',
            # with rotating handlers you can configure max size and other stuff
            'class': 'logging.FileHandler',
            'formatter': 'simple',
            'filename': 'my_log.log',
        },
    # that's the most interesting part
    'loggers':
        'django': {
            'handlers': ['file', 'console'],
            'level': 'INFO',
        },
        'celery': {
            'handlers': ['file', 'console'],
            'level': 'INFO',
        },
}

Note: your logging configuration should be more complex.

Now, no further action is required, given the default django celery setup from the docs, and that your django version is 4.2.11, and celery is 5.3.6 - on the older versions you might want to play with celery.signals.setup_logging.

2nd note: streams are set up for separate django and celery processes: you will not see them all in one place

3. Consider using tools like logstash, sentry, graypy, new relic - they will gather all logs into one place. I haven't tried neither one of them, but it's something that is also used.

Lastly, of course this could work with anything, not necessarily just django.