I want to keep a long process running in the background in django

Question

I am a beginner in django.

I want to keep running long processes in django in the background.

And I want to keep it running unless I explicitly end that process.

I can't figure out where and how to add the following code to django.

import threading
import asyncio


class long_task:
    def __init__(self):
        self.continue_flag = True

    async def long_task(self,):
        print('long task start...')
        i = 0
        while self.continue_flag:
            print(i)
            await asyncio.sleep(3)
            i = i+1

    def stop(self):
        self.continue_flag = False


def run_loop(loop):
    asyncio.set_event_loop(loop)
    print('main loop start...')
    loop.run_forever()


def ask_loop_stop(loop):
    print("main loop stop")
    loop.call_soon_threadsafe(loop.stop)


loop = asyncio.get_event_loop()
threading.Thread(target=run_loop, args=(loop,)).start()
print('main loop is ready')

long_task_1 = long_task()
asyncio.run_coroutine_threadsafe(long_task_1.long_task(), loop)

I will elaborate on what I want to do.

1. Run a loop when django starts.
2. Add long_task to the loop at any given time (when a specific event occurs by monitoring an external site).
3. Keep the program running in the background so that the site can be viewed. In the future I would like to display the results of this program on a page.
4. Close the loop at another time (during maintenance).

Here's what I've tried

1. Register and execute as a task using celery and celery-beat.

=>

Can't specify that a program should be run when django starts

2. Create a custom task in django and run it with a command.

.

class Command(BaseCommand):
    help = "Start Main Loop"

    def add_arguments(self, parser):
        parser.add_argument('-i', default=1, type=int, help='UserId')

    def handle(self, *args, **options):
        id = options['i']
        print("UserId:{},".format(id))

        loop = asyncio.get_event_loop()
        threading.Thread(target=run_loop, args=(loop,)).start()
        print('main loop is ready')

=> I get an following error and can't run it.

”django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.”

I also don't know what I can do when I start django.

3. Add the code to init.py.

.

import asyncio
import threading
import time
from app import long_task

loop = asyncio.get_event_loop()
threading.Thread(target=long_task.run_loop, args=(loop,)).start()
print('main loop is ready')

=>

I get an following error and can't run it.

”You cannot call this from an async context - use a thread or sync_to_async.”

--Environment--

• Windows10 64bit
• Python (3.7)
• Redis server (3.0.504)
• Django (3.2.3)
• celery (5.0.5)
• django-celery-beat (2.2.0)
• django-celery-results (2.0.1)
• PostgreSQL (13)

Please tell me how to make this happen.

Answer 1

AsyncIO event loops are not thread safe; you can't run the loop from a different thread than it was originally created on. Your run_loop function should instead take no arguments, and create/start a new event loop to run your coroutine:

LOOP = None

def run_loop():
    global LOOP
    LOOP = asyncio.new_event_loop()
    LOOP.run_until_complete(long_running_task())

threading.Thread(target=run_loop).start()

# <do other things>

LOOP.call_soon_threadsafe(LOOP.stop)