I am looking to run queued jobs using RQ, but looking at the following example:
from rq import Queue
from redis import Redis
from somewhere import count_words_at_url
# Tell RQ what Redis connection to use
redis_conn = Redis()
q = Queue(connection=redis_conn) # no args implies the default queue
# Delay execution of count_words_at_url('http://nvie.com')
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job.result # => None
# Now, wait a while, until the worker is finished
time.sleep(2)
print job.result # => 889
I see time.sleep(2) - and I was wondering if this is mandatory to specify. The jobs I schedule could take (at times) an hour to complete (this varies on a per job basis).
Is RQ still suitable for such jobs - where execution time vastly varies?
Any suggestions would be great!
As long as each job you schedule gives some clear, observable indication that it's all done, you can definitely use RQ and wait for such "indications"; then you'll rely on such indications to be able to tell when to access each job's result.
In the example you quote, it's apparently assumed that count_words_at_url gives no clear indication about when it's finished (you could "poll" by looping on while job.result is None: with a sleep in the while's body, but that's pretty fragile -- avoid polling when you can).