Search code examples
rabbitmqcelery

Run client channel backup poller Celery Beat

I am working on a project with Celery Beat and Worker in Kubernetes.

I ran a project with this config.

app = Celery('celery-worker',
             broker=RABBITMQ_URL,
             backend=REDIS_URL
             )
app.conf.update(
    result_expires=3600,
)

And the run command is:

celery -A app worker -B -l INFO

After running it, the Celery Beat log shows a lot of lines: backup_poller.cc:138] Run client channel backup poller: UNKNOWN:pollset_work {created_time:"2022-12-10T15:25:01.080085021+03:30", children:[UNKNOWN:Bad file descriptor {created_time:"2022-12-10T15:25:01.080072267+03:30", errno:9, os_error:"Bad file descriptor", syscall:"epoll_wait"}]}

After looking for some solutions, I changed the pool option in the Celery run command into --pool=gevent, but that did not work either.

how can I solve the problem?

Solution

  • Based on Document, For production environment, you need to start celery beat separately. So you need separate pods for ‌Beat and Worker and use --pool only for Worker pod.

    celery -A app beat -l INFO
celery -A app worker -l INFO --pool=gevent -c 1