How do I make a method running subprocess command, asynchronous?

Question

Problem:

• Using Celery and Redis to make a subprocess call asynchronous, which is not working as expected.
• Django Function-based View is making a call to the function which runs the subprocess command, and the API request is waiting for the subprocess command to complete to return the response.

The below method is called through an function-based views in django. I have made this method a shared_task of 'celery' and the message broker used is 'Redis'.

import subprocess
from celery import shared_task

@shared_task
def run_docker_command(command):
    try:
        process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
        output, error = process.communicate()
        print("Executed!!")
        if process.returncode == 0:
            print("Command executed successfully.")
            if output:
                print("Output:\n", output.decode())
            return 1
        else:
            print("Command execution failed.")
            if error:
                print("Error:\n", error.decode())
            return 0

    except OSError as e:
        print("Error executing the command:", str(e))

When the below endpoint is called (with both django and celery server running), the endpoint start_container is waiting until the process inside the run_docker_command is completed to give the response.

@csrf_exempt
def start_container1(request):
    if request.method == 'POST':
        from Visiotwin_Configuration.tasks import run_docker_command
        command = "az container start -n <container-name> -g <res-group-name>"
        try:
            task = run_docker_command.apply_async(args=[command])
            print(task)
            res = task.get()
            if res == 1:
                status = "Docker started successfully"
            else:
                status = "Docker failed to start"
        except:
            status = "Something went wrong!"

        return JsonResponse(status, safe=False)

Hope the problem statement is clear.

Any suggestions and changes are open. Thank you!

Answer 1

AsyncResult.get() waits for the task till it's completed and returns the response. If you want your API to return immediately, remove .get() and the status will always be pending.

@csrf_exempt
def start_container1(request):
    if request.method == 'POST':
        from Visiotwin_Configuration.tasks import run_docker_command
        command = "az container start -n <container-name> -g <res-group-name>"
        task = run_docker_command.apply_async(args=[command])

        return JsonResponse(task.status, safe=False)