I have created a Query resource in rails. When a Query is POSTed, I create Query record and fire 2 task workers in the background using Sidekiq.
My query model in addition to having the query parameters also contains status for each of the task workers. At the time of Query creation, the status for each task is set to "created".
@query = Query.create(param1: parameter1, param2: parameter2, task1_status: "created", task2_status: "created")
Task1Worker.perform_async(@query.id)
Task2Worker.perform_async(@query.id)
Task1 worker uses the query parameters to perform some processing and based on the results updates the task_status to either "completed" or "failed"
q = Query.find(query_id)
status = process_task1()
if status == SUCCESS
q.task1_status = "completed"
else
q.task1_status = "failed"
end
q.save!
Task2 worker waits for Task1 to be completed before starting processing
q = Query.find(query_id)
count = 1
while q.task1_status == "created"
if count == 3
logger.error("Task1 state change timed out")
return
end
sleep(5)
q = Query.find(query_id)
end
status = process_task2()
if status == SUCCESS
q.task2_status = "completed"
else
q.task2_status = "failed"
end
q.save!
Task 1 needs about 4 secs for the processing and sets the state to either "completed" or "failed". But Task 2 never sees the update & times out after 10 secs, without processing. Is there something I am missing here. Is there a better way to do this?
But Task 2 never sees the update & times out after 10 secs, without processing
I'm suspecting caching problems (Query.find(query_id) hitting cache and not database). Try using q.reload instead of q = Query.find(query_id) on each retry.
Or, much better, schedule Task2 from Task1 when it completes, so that you don't have to tie your worker for long periods of time doing nothing.