I have some tasks that I would like to run in a parallel fashion. They are almost 100% IO bound and not cpu intensive (simply making API requests).
I use celery/celerybeat in a Docker environment and have just added --pool=gevent to my docker-compose file, but I must also pass --beat.
When I start my app I get the following error:
celery.exceptions.ImproperlyConfigured: -B option doesn't work with eventlet/gevent pools: use standalone beat instead.
Is there any way to use thread execution with Celery beat?
What you are trying to do (i.e. embed beat inside the worker) isn't recommended anyway. So I'd suggest you start a standalone Celery beat (as adviced by the error message) and worker(s) with gevent pool. The worker's part is where you take advantage of gevent as they execute the I/O bound tasks, not the Celery beat that just kicks them off.
Alternatively, if you use Celery 4.4 or newer, you might try to use --pool threads to you threaded task pool.