Is there a way to simultaneously write the result of each joblib.parallel run into its own file?

Question

Each "parallel" result of a single parallel run needs to be written into it's own file. This can also be solved if I were able to name each result.

I have a function that generates some data. Each time it is run, the data differs slightly and therefore I need to run it a few times. I currently have working code that uses joblib.Parallel to speed up this process. The issue is that the result is one long list of all the parallel runs and to write this into separate files is complicated and error-prone.

def fn(x):
    for i in np.linspace(0, x, 1000):
        a = x
        b = 2*x
        return a, b

ans = Parallel(n_jobs=-1)(delayed(fn)(x) for x in np.linspace(0,5,5))
ans
# I need to either name/extract each result in the list below, or directly write each into its own file
out[]: [(0.0, 0.0), (1.25, 2.5), (2.5, 5.0), (3.75, 7.5), (5.0, 10.0)]

Answer 1

If you just want each process to write to it's own file you can do the following.

def fn(x):
    for i in np.linspace(0, x, 1000):
        a = x
        b = 2*x
        with open(str(x)+"_file.csv", 'w') as file:
            file.write(a, b)

        return a, b

ans = Parallel(n_jobs=-1)(delayed(fn)(x) for x in np.linspace(0,5,5))

But I am not sure why you would want to do this, if you let us know what is your end goal in more detail I am sure we can help more.