I want to break down a project to small microservices. I've been thinking of different tools like gRPC, but I think celery might be a better option for me. Please if I'm wrong and celery is not a good choice, tell me why.
What exactly I want to do?
- For example, I have three different servers, A, B and C. "A" must be able to send a request to "B". "B" must be listening all the time.
- "B" get request from "A", works on it, and then it must be able to send request to "C".
- "C" does some stuff with the given request from "B" and then it must return the result to "B".
- "B" works on the result and when it's finished, it must return the result to "A".
What have I done?
- I created two RabbitMQ servers, one for "A" & "B" and another one for "B" and "C".
- I used send_task to send tasks to servers. One task to send a string from "A" to "B". "B" is already listening and waiting for requests from "A". There's another send_task on "B" which sends a request to "C". "C" is also already running and waiting for requests from "B".
- I used backend='rpc' to return results.
When a request is sent by "A", "B" receives the request and send_task to "C". "C" also receives the request with no problem. The problem is returning results.
I tried different things, but I mostly got runtimeError: Never call result.get() within a task Celery error.
Does anyone have any solution?
Here I'm answering to my own question,
In my case this answer didn't work. So I read a bit more and I updated the code like below :
timeout = 0.1
while not ML_task.ready():
time.sleep(timeout)
result = ML_task.get(disable_sync_subtasks=False)
Hope it helps.