Python Multiprocessing Return Value Processing

Question

I'm having trouble on how to do a return results comparison for each of the multiprocessing. I am doing a multiprocessing for my function. My function will return a value. I want to run my function 5 times and compare the which process have the lowest return value. My code as below.

def do_processVal():
   getParamInit()
   do_evaluation()
   currbestVal = bestGlobalVal

   return 'Current best value: ', currbestVal, 'for process{}'.format(os.getpid())

from multiprocessing import Pool
import concurrent.futures
from os import getpid
import time
import os

start = time.perf_counter()

with concurrent.futures.ProcessPoolExecutor() as executor:
   results = [executor.submit(do_processVal) for _ in range(5)]

   for f in concurrent.futures.as_completed(results):
      print(f.results())

finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')

Output as of now:

Current best value: 12909.5 for process 21918
Current best value: 12091.5 for process 21920
Current best value: 12350.0 for process 21919
Current best value: 12000.5 for process 21921
Current best value: 11901.0 for process 21922
Finish in 85.86 second(s)

What I want is from all the 5 return values above, I want to take the data for value that is the lowest. In this example process 21922 have the lowest value. So I want to assign the value to a parameter.

FinalbestVal = 11901.0

Answer 1

From what I see, you are mixing the responsibilities of your function and that is causing your issues. The function below returns only the data. The code that is run after the data is collected evaluates the data. Then the data is presented all at once. Separating the responsibilities of your code is a decades old key to cleaner code. The reason it has been supported as a best practice for so long is because of problems like the one you have run into. It also makes code easier to reuse later without having to change it.

def do_processVal():
   getParamInit()
   do_evaluation()
   currbestVal = bestGlobalVal
   return [currbestVal, os.getpid()]

from multiprocessing import Pool
import concurrent.futures
from os import getpid
import time
import os

start = time.perf_counter()

with concurrent.futures.ProcessPoolExecutor() as executor:
    results = [executor.submit(do_processVal) for _ in range(5)]

best_value = -1
values_list = []
for f in concurrent.futures.as_completed(results):
    values = f.result()
    values_list.append(values)
    if best_value == -1 or values[0] < best_value:
        best_value = values[0]

for i in values_list:
    print(f'Current best value: {i[0]} for process {i[1]}')

finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
print(f'Final best = {best_value}')