Search code examples
pythonpython-3.xpython-multiprocessing

Will python multiprocessing returns the same sequence as for loop?


I have a for loop looks like below:

function1_values = [function1(solution[i])for i in range(0,pop_size)]

I am trying to use multiprocessing to save some computation time. My idea is like below

cores = multiprocessing.cpu_count() - 1
pool = multiprocessing.Pool(processes=cores)
function_values = pool.map(function1, solution[:len(pop_size))
pool.close()
pool.join()

I am wondering will the multiprocessing returns the same output as the first command? Especially, will the sequence of the results is the same?


Solution

  • Yes, the order of multiprocessing.pool.Pool.map() is ordered, with the results in the same sequence.

    Same for the imap() variant, which returns the results as they're ready, but still guarantees the correct order. Results later in the sequence which become ready "too soon" will be held back until they can be returned in the correct sequence position.

    If you don't care about the sequence, there's a variant method, imap_unordered(), which not only returns the results as they're ready but also potentially returns them out of sequence, whichever ones complete first; this would be useful if there is variance in how long each result takes to calculate and the results don't have any particular ordering to them anyway (and/or have tags indicating which one's which).