Django Celery Rate Limit setting doesn't work as expected

Question

I use below command to create one worker:

celery -A proj worker -l info --concurrency=50 -Q celery,token_1 -n token_1

And in my task, I set the rate limit to 4000/m.

However, when I start running the collection, I noticed the average task processed is just around 10-20/s (with rate limit rule 4000/m enabled). Then, I removed the rate limit rule, now the task rates goes to around 60/s.

I am confused, since my rate limit is 4000/m, which is relatively 65/s. Why it finally goes just 10-20/s????? (I have already set 50 threads for the worker....)

Answer 1

You're misunderstanding how the rate limits operate in celery. 'According to the documentation on version 4.2:

The rate limits can be specified in seconds, minutes or hours by appending “/s”`, “/m” or “/h” to the value. Tasks will be evenly distributed over the specified time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of 600ms between starting two tasks on the same worker instance.

In essence, celery was adding a forced delay between your tasks. Since each task was already processing in about 16ms (1/60 of a second), adding another 16 ms forced delay between tasks reduced the rate at which they would process.