Search code examples
pythondjangomod-wsgiwagtailpython-huey

Huey ``db_task`` successfully registered by consumer but does not receive/execute any task

I have a Django project with huey as task queue. In my dev environment (see below), everything works as expected:

  1. [save] Django model instance saved
  2. [trigger] huey task triggerd
  3. [execute] huey task gets executed by running consumer (run_huey)

But I could not get the third point [execute] working in my production environment. The only difference I can spot is that and the consumerrun_huey is started by a systemd service unit and the webserver is an Apache2 (see below). It does not matter if I set DEBUG = True in my production environment.

The consumer does correctly recognize the db_task "schedule_virusscan", but does not pick up/receive any task triggered by my model save method (here: schedule_virusscan(self.id))

I use immediate=False for my huey instance, as I run run_huey even in my dev environment.

I get this behavior for huey.FileHuey and huey.SqliteHuey.

Question

What am I missing that my consumer (only on production) does not execute or even receive any task?

Setup

Webserver

  • apache (2.4) with mod_wsgi (4.6.5) on Debian Buster
  • run_huey and the WSGIDaemonProcess do run as the same (dedicated, non-root) user

Django project

# settings.py

DEBUG = False  # True in dev environment

HUEY = {
    'huey_class': 'huey.FileHuey',
    'path': '/tmp/huey',
    'immediate': False,
    'immediate_use_memory': False,
    'consumer': {
        'workers': 2,  # use 2 threads
    },
}

# my-huey-systemd-service-unit.service

[Unit]
After=network.target

[Service]
Restart=always
WorkingDirectory=/path/to/project
ExecStart=/path/to/venv/bin/python3 \
    /path/to/project/manage.py run_huey \
    --logfile=/path/to/project/wagtailapiforms-huey.log
ExecReload=/bin/kill -s SIGHUP $MAINPID
ExecStop=/bin/kill -s SIGINT $MAINPID

[Install]
WantedBy=default.target

# project/apps/core/tasks.py

from huey.contrib.djhuey import db_task

@db_task()
def schedule_virusscan(attachment_id):
    print(f"Scan for virus for attachment with pk {attachment_id}...")
    ...

# project/apps/core/models.py

class Attachment(models.Model):
    ...
    def save(self, *args, **kwargs):
        from .tasks import schedule_virusscan
        super().save()
        print(f"Attachment instance {self.id}: trigger schedule_virusscan...")
        _scan = schedule_virusscan(self.id)
        # _scan()

# prints from server

[Tue Jun 29 2021] [wsgi:error] [pid 4825] [remote 10.1.x.x:36754] Attachment instance created, calling super().save()...
[Tue Jun 29 2021] [wsgi:error] [pid 4825] [remote 10.1.x.x:36754] Attachment instance 6b0dd19d-377d-43b6-9d9a-343c80793447: trigger schedule_virusscan...
[Tue Jun 29 2021] [wsgi:error] [pid 4825] [remote 10.1.x.x:36754] _scan:   <Result: task 22306b18-d45e-4458-8052-138be9c996f1>

# prints from run_huey
# /path/to/project/wagtailapiforms-huey.log

[2021-06-29] INFO:huey.consumer:MainThread:Huey consumer started with 2 thread, PID 4494 at 2021-06-29 18:48:06.834951
[2021-06-29] INFO:huey.consumer:MainThread:Scheduler runs every 1 second(s).
[2021-06-29] INFO:huey.consumer:MainThread:Periodic tasks are enabled.
[2021-06-29] INFO:huey.consumer:MainThread:The following commands are available:
+ project.apps.core.tasks.schedule_virusscan

# Here I expected the output of the execution of received tasks - but on production this consumer does not receive anything.
# In my development environment this looks like:
# [2021-06-29 INFO:huey:Worker-2:Executing project.apps.core.tasks.schedule_virusscan: 4f877b2e-559a-43c9-9bf3-5821ea426842
# Huey task schedule_virusscan: Scan for virus for attachment with pk 38950ef9-469a-485c-afee-24cbc6fcef12...
# [2021-06-29] INFO:huey:Worker-2:project.apps.core.tasks.schedule_virusscan: 4f877b2e-559a-43c9-9bf3-5821ea426842 executed in 0.109s
Solution

  • The problem had nothing to do with huey or mod_wsgi, but with the location of the huey SqliteHuey or FileHuey path setting and the apache2 systemd service unit with PrivateTmp=true (see /lib/systemd/system/apache2.service, assuming Debian 10).

    Solution: Locate huey database/file not below global /tmp, but in a separate directory, to which the consumer (run_huey) as well as the webapp has access.

    The implications of PrivateTmp=true (just a selection):