I have a Django application that needs to access web APIs that are rate-limited by API key. To manage requests to the API, I've created a simple background server that polls my Django ORM for enqueued task requests, and executes them asynchronously with respect to my app servers (the app servers don't need to block on the results). The worker server is launched using a custom manage.py command. The processing requirements of these external lookups are extremely minimal, so I want to run them from my main web service, directly. I have already rigged it with a makeshift mutex, so that when my application scales, only one of these worker servers will run.
I am trying to figure out the best way to automatically launch this worker server when I deploy my application. I tried adding it as a command to my supervisord.conf file, but this failed with the error ImportError: No module named django.core.management. I assume this is because supervisord isn't running in the virtual environment in which Django is installed. So then, I tried running a separate copy of supervisord from the postinstall hook. This is causing my deployment to fail because postinstall waits for all commands to complete before terminating, and it is timing out.
Is there a way to fix either of these approaches, or another approach that might work? It seems to me that I'll run into similar problems even if I do create a separate python-worker service.
You are on the right track by putting it into supervisord.conf. You are also right on why you are getting your error about not finding Django. Luckily this is an easy fix.
You just need to make sure you call your django manage.py with the correct python binary (the one inside of the virtualenv), and it should work fine for you.
Here is an example using fully qualified paths to make sure we are using the right files. Your path to manage.py will most likely be different unless your app is also called myapp, and you will need to change the my_awesome_command to the name of your custom command.
[program:custom_command]
directory = /home/dotcloud/current/
command = /home/dotcloud/env/bin/python /home/dotcloud/current/myapp/manage.py my_awesome_command
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log