Search code examples
pythonparallel-processingconcurrent.futures

Parallelize a simple loop in Python and get results with concurrent.futures

Let suppose I have complicated function I cant to run on a list :

import concurrent.futures
import random
import numpy as np    

inputData = random.sample(range(60000, 1000000), 50)

def complicated_function(x):
"""complicated stuff"""
return x**x

I can process directly in a loop this way (that are how I want the results at the end of my code, with the same order if possible) :

#without parallelization
processedData = [complicated_function(x) for x in inputData]

for parallel processing I look at tutorials and it I made this code :

#with parallelization
processedData2 = []
with concurrent.futures.ThreadPoolExecutor() as e:
    fut = [e.submit(complicated_function, inputData[i]) for i in range(len(inputData))]
    for r in concurrent.futures.as_completed(fut):
        processedData2.append(r.result())

The problem is that watching at my system monitor there is just one core working when this is running.. so obviously this is not working as I need...

Thank a lot in advance for your help !

Solution

  • It is because you are threading, and pythons global interpreter lock doesnt actually run the tasks in parallel across multiple cores, when you use threading.

    Instead u can use multiprocessing..Just like ThreadPoolExecutor, there is a ProcessPoolExecutor which will make sure multiple cores are utilised.