I am developing my final degree project and I am facing some problems with Python, Flask, socketIO and background threads.
My solution takes some files as input, process them, makes some calculations, and generates an image and a CSV file. Those files are then uploaded to some storage service. I want to make the processing of the files on a background thread and notify my clients (web, Android, and iOS) using websockets. Right now, I am using flask-socketIO with eventlet as the async_mode of my socket. When a client uploads the files, the process is started in a background thread (using socketio.start_background_task) but that heavy process (takes about 30 minutes to end) seems to take control of the main thread, as a result when I try to make an HTTP request to the server, the response is loading infinitely.
I would like to know if there is a way to make this work using eventlet or maybe using another different approach.
Thank you in advance.
Eventlet uses cooperative multitasking, which means that you cannot have a task using the CPU for long periods of time, as this prevents other tasks from running.
In general it is a bad idea to include CPU heavy tasks in an eventlet process, so one possible solution would be to offload the CPU heavy work to an external process, maybe through Celery or RQ. Another option that sometimes works (but not always) is to add calls to socketio.sleep(0) inside your CPU heavy task as frequently as possible. The sleep call interrupts the function for a moment and allows other functions waiting for the CPU to run.