Multithreading makes me get the "ValueError: I/O operation on closed file" error. Why?

Question

I am writing a Flask Web Application using WTForms. In one of the forms the user should upload a csv file and the server will analyze the received data. This is the code I am using.

filename = token_hex(8) + '.csv'  # Generate a new filename
form.dataset.data.save('myapp/datasets/' + filename)  # Save the received file
dataset = genfromtxt('myapp/datasets/' + filename, delimiter=',')  # Open the newly generated file
# analyze 'dataset' 

As long as I was using this code inside a single-thread application everything was working. I tried adding a thread in the code. Here's the procedure called by the thread (the same exact code inside a function):

def execute_analysis(form):
    filename = token_hex(8) + '.csv'  # Generate a new filename
    form.dataset.data.save('myapp/datasets/' + filename)  # Save the received file
    dataset = genfromtxt('myapp/datasets/' + filename, delimiter=',')  # Open the newly generated file
    # analyze 'dataset'

and here's how I call the thread

import threading

@posts.route("/estimation", methods=['GET', 'POST'])
@login_required
def estimate_parameters():
    form = EstimateForm()
    if form.validate_on_submit():
        threading.Thread(target=execute_analysis, args=[form]).start()
        flash("Your request has been received. Please check the site in again in a few minutes.", category='success')
        # return render_template('posts/post.html', title=post.id, post=post)
    return render_template('estimations/estimator.html', title='New Analysis', form=form, legend='New Analysis')

But now I get the following error:

ValueError: I/O operation on closed file.

Relative to the save function call. Why is it not working? How should I fix this?

Answer 1

It's hard to tell without further context, but I suspect it's likely that you're returning from a function or exiting a context manager which causes some file descriptor to close, and hence causes the save(..) call to fail with ValueError.

If so, one direct fix would be to wait for the thread to finish before returning/closing the file. Something along the lines of:

def handle_request(form):
  ...
  analyzer_thread = threading.Thread(target=execute_analysis, args=[form])
  analyzer_thread.start() 
  ...
  analyzer_thread.join() # wait for completion of execute_analysis
  cleanup_context(form)
  return

Here is a reproducable minimal example of the problem I am describing:

import threading

SEM = threading.Semaphore(0)

def run(fd):
    SEM.acquire() # wait till release
    fd.write("This will fail :(")

fd = open("test.txt", "w+")
other_thread = threading.Thread(target=run, args=[fd])
other_thread.start()

fd.close()
SEM.release() # release the semaphore, so other_thread will acquire & proceed
other_thread.join()

Note that the main thread will close the file, and the other thread will fail on write call with ValueError: I/O operation on closed file., as in your case.