parallel computing combination_with_replacement using multiprocessing

Question

I'm trying to get the all possible combination with replacement and make with each of them some calculation. I'm using the code below:

from itertools import combination_with_replacement

for seq in combination_with_replacement('ABCDE', 500):
    # some calculation

How can I parallelize this calculation using multiprocessing?

Answer 1

You can use the standard library concurrent.futures.

from concurrent.futures import ProcessPoolExecutor
from itertools import combinations_with_replacement

def processing(combination):
    print(combination)
    # Compute interesting stuff


if __name__ == '__main__':
    executor = ProcessPoolExecutor(max_workers=8)
    result = executor.map(processing, combinations_with_replacement('ABCDE', 25))
    for r in result:
         # do stuff ...

A bit more explanations:

• This code creates an executor using processes. Another possibility would be to use threads but full python threads only run on one core so it might not be the solution of interest in your case as you need to run heavy computation.
• The map object return a asynchronous object. Thus, the line executor.map.. is non blocking and you can do other computation before collecting the result in the for loop.
• It is important to declare the processing function out of the if __name__ == '__main__': block and to declare and use the executor in this block. This prevent for infinite executor spawning and permit to pickle the worker function to pass it to the child process. Without this block, the code is likely to fail.

I recommend this over multiprocessing.Pool as it has a more clever way to dispatch the work as you are using an iterator.

Note that your computation for combination of 500 with 5 elements ABCDE might not be possible. It needs to compute 5**500 > 1e350 elements. By parallelizing, you will only reduce your computation linearly by a factor max_workers, so in this case 8 and each process will need to run with ~ 1e349 elements, which should take about ~ 1e335 years if each computation is done in 1 micro second.