multiprocessing result to list

Question

I want to let my results of the multiprocessing to 2 lists how could I do that. I tried below but only get the last one list.

def multiply(x, y):
  return x * x, x * y

#use wrap
def multiply_wrapper(args):
  return multiply(*args)

#use wrap
if __name__ == "__main__":
  list_x = []
  list_y = []

  p = Pool(4)
  values = [(x, y) for x in range(4) for y in range(4)]
  results = p.map(multiply_wrapper, values)
  
  list_x.append(results[0][0])
  list_y.append(results[0][1])

  print(list_x)
  print(list_y)


  

Answer 1

I tried your code with small changes and results looks ok.

Did you perhaps expect the multiprocessing to leave the data separated according to which worker processed the chunk of data? Fortunately, it joins the data for us.

The following inspired from docs and your original question. Also don't forget to release resources as the garbage collector will not do this for you.

from multiprocessing.pool import Pool
import pprint

def multiply(x, y):
  return x * x, x * y

#use wrap
def multiply_wrapper(args):
  return multiply(*args)

#use wrap
if __name__ == "__main__":
  p = Pool(4)
  values = [(x, y) for x in range(4) for y in range(4)]
  results = p.map(multiply_wrapper, values)
  p.close()
  p.join()
  pprint.pprint(results)

Output:

[(0, 0),
 (0, 0),
 (0, 0),
 (0, 0),
 (1, 0),
 (1, 1),
 (1, 2),
 (1, 3),
 (4, 0),
 (4, 2),
 (4, 4),
 (4, 6),
 (9, 0),
 (9, 3),
 (9, 6),
 (9, 9)]

---

Replying to comments:

To obtain 2 lists from the output of Pool.map, we can unzip the result.

if __name__ == "__main__":
    p = Pool(4)
    values = [(x, y) for x in range(4) for y in range(4)]
    results = p.map(multiply_wrapper, values)
    p.close()
    p.join()

    list_x = [x[0] for x in results]
    list_y = [x[1] for x in results]
    pprint.pprint(list_x)
    pprint.pprint(list_y)

Output:

[0, 0, 0, 0, 1, 1, 1, 1, 4, 4, 4, 4, 9, 9, 9, 9]
[0, 0, 0, 0, 0, 1, 2, 3, 0, 2, 4, 6, 0, 3, 6, 9]