Search code examples
pythonnumpymultiprocessingpoolioerror

Multiprocessing IOError: bad message length


I get an IOError: bad message length when passing large arguments to the map function. How can I avoid this? The error occurs when I set N=1500 or bigger.

The code is:

import numpy as np
import multiprocessing

def func(args):
    i=args[0]
    images=args[1]
    print i
    return 0

N=1500       #N=1000 works fine

images=[]
for i in np.arange(N):
    images.append(np.random.random_integers(1,100,size=(500,500)))

iter_args=[]
for i in range(0,1):
    iter_args.append([i,images])

pool=multiprocessing.Pool()
print pool
pool.map(func,iter_args)

In the docs of multiprocessing there is the function recv_bytes that raises an IOError. Could it be because of this? (https://python.readthedocs.org/en/v2.7.2/library/multiprocessing.html)

EDIT If I use images as a numpy array instead of a list, I get a different error: SystemError: NULL result without error in PyObject_Call. A bit different code:

import numpy as np
import multiprocessing

def func(args):
    i=args[0]
    images=args[1]
    print i
    return 0

N=1500       #N=1000 works fine

images=[]
for i in np.arange(N):
    images.append(np.random.random_integers(1,100,size=(500,500)))
images=np.array(images)                                            #new

iter_args=[]
for i in range(0,1):
    iter_args.append([i,images])

pool=multiprocessing.Pool()
print pool
pool.map(func,iter_args)

EDIT2 The actual function that I use is:

def func(args):
    i=args[0]
    images=args[1]
    image=np.mean(images,axis=0)
    np.savetxt("image%d.txt"%(i),image)
    return 0

Additionally, the iter_args do not contain the same set of images:

iter_args=[]
for i in range(0,1):
    rand_ind=np.random.random_integers(0,N-1,N)
    iter_args.append([i,images[rand_ind]])

Solution

  • This is what solved the problem: declaring the images global.

    import numpy as np
    import multiprocessing
    
    
    N=1500       #N=1000 works fine
    
    images=[]
    for i in np.arange(N):
        images.append(np.random.random_integers(1,100,size=(500,500)))
    
    def func(args):
        i=args[0]
        images=images
        print i
        return 0
    
    iter_args=[]
    for i in range(0,1):
        iter_args.append([i])
    
    pool=multiprocessing.Pool()
    print pool
    pool.map(func,iter_args)