Search code examples
pythonnestjsmicroservicesmessage-queuetask-queue

Architecture and interaction of the client, NestJS backend, and Python microservice

As the title says, I have a question about the architecture and interaction of the client, NestJS backend, and Python microservice. I need to develop a recognition service. I have a client application that needs to send an HTTP request with an image to the NestJS backend, which should contact a third-party Python microservice, this service will recognize the text from the image (this can take a long time) and we should receive this response on the client application. What is the best way to implement this? I have an idea to connect NestJS with a Python microservice via RabbitMQ, the client sends a POST HTTP request to the NestJS backend, the backend sends a message to create a task via RPC to the Python microservice, which creates a Redis RQ queue with the task, returns the backend to the NestJS id of the task and starts to perform a long-running task, we return the task id to the client from the NestJS backend. After that, the client, with a certain interval, can send a GET HTTP request to the NestJS backend, which in turn will send a message to get the status of the task to the microservice and return the status to the client. Here is a rough sketch of how I think it should work Is this a good way to do it or is it possible to somehow optimize this process or implement it more competently?

Solution

  • I think you're on the right track here.

    1. Send image to nest via HTTP - yes
    2. Post job to redis queue - yes, use nestjs builtin queue handling (see docs), this will make it easier to consume the product of the job as well
    3. Instead of having your client poll for a result, check out Server-sent Events

    Server sent events are intended exactly for the use-case you are using.