Many popular task queues (such as Google GAE TaskQueue, Celery) have the ETA/Countdown feature, which allows a task to be put into the queue after xxx seconds.
I am working on a project that needs a task queue with the ETA feature. However, there are some limitations that I have to use the Google Pubsub messaging system. Pubsub does not have the ETA feature. I am wondering how to implement a reliable and scalable ETA mechanism for a task queue. Both general architecture ideas and actual code samples are welcome.
Our system enqueues 600-2000 tasks/second, and about 10% of them need to have ETA. It is a distributed system and performance-critical.
I tried to trace the source code of celery, but couldn't find the actual logic of handling the ETA. It would also be good if someone can point me to the file/code of Celery that handle ETA.
I think I might have found how Celery did it. In eventlet.py, it uses eventlet's spawn_after feature to delay the worker creation "ETA" seconds.
secs = max(eta - monotonic(), 0)
g = self._spawn_after(secs, entry)