I am using celery to perform some time triggered tasks. Using normal command line methods I am easily able to generate tasks and execute them. For this I have to create two daemons, one for celery beat and another for celery worker.
Now though celery does solve my issue of creating and executing time triggered tasks, but it comes at the cost of two extra daemons for my system. I already have a daemon running for my module and I want to reuse it (add celery worker and beat as a thread to the existing module) . Is there a way to do so?
You can use app.worker_main() to run the worker as a thread.
Since you were able to run it from command line, I assume you have already built an object for the Celery class(usually termed as app in docs). You can try the following code to run the worker as a thread.
from celeryconfig import app # config file you must have build earlier where app = Celery(), object of Celery class
def worker():
# Arguments you give on command line
argv = [
'worker','-A','<module>.tasks',
'-P','gevent', # Would probably need this argument if running with other Greenlets
'--loglevel=info']
app.worker_main(argv)
Just create a thread for the above method in you main.py file of the module.
For celery beat, I had earlier tried building a similar method myself. (Not sure if there exists a method already). I wrote the following method and added it to the Celery class
def beat_main(self, argv=None):
return instantiate(
'celery.bin.beat:beat',
app=self).execute_from_commandline(argv)
Your Celery class is written in /usr/local/lib/python2.7/dist-packages/celery/app/base.py. Try using it the same way as the worker (app.beat_main with arguments). Hope it works for you as well.