MemoryError in Pickle while using Multiprocessing

Question

I am running monte carlo simulations of some physical system and I need to increase the granularity of the simulation timesteps (so n_timesteps=t/dt where t is fixed). Which obiviously leads to a longer runtime for each simulation (as well as to an increased memory usage). Because of this I changed my code to make use of all my processors using pythons multiprocessing. This works fine for coarse simulations (larger dt) but as soon as I decrease dt I run into a pickling error:

multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x000000000B5D6EC8>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'

I use a package which basically does all the simulations. So my code looks similar to this:

import numpy as np
import multiprocessing as mp
from specific_package import system_simulator

def subsimulation(parameter, other_parameters):
    n_montecarlo = 2
    simulator = system_simulator(other_parameters, n_montecarlo)
    # parameter is an numpy array of shape(n_timesteps, 13)
    simulator.set_parameters(parameter)  # At this point the error gets thrown
    results = simulator.calculate()
    # results will be a list of length n_montecarlo with each element contatining a list of length 
    # n_timesteps each containg a (4, 4) matrix (in a for the package specific dataformat)
    # results is then reduced to 4 numpy arrays of length n_timesteps each:
    a, b, c, d = np.ones(n_timesteps), np.ones(n_timesteps), np.ones(n_timesteps), np.ones(n_timesteps)
    return [a, b, c, d]


def run_simulation(foo):
    # does some stuff then run the simulations in parallel:
    runs = [i+1 for i in range(96)]  # As in total I want to have ~200 single simulations
    pool = mp.Pool(mp.cpu_count())
    results = pool.starmap_async(subsimulation, [(parameter, other_parameters) for run in runs]).get()
    pool.close()
    avg_res = np.mean(np.array(results), axis=0)
    return [avg_res[0], avg_res[1], avg_res[2], avg_res[3]]

This for example does not work for n_timesteps = 60000, which I think should actually be manageable. I already reduced the number of simulations in each subsimulation to 2 such that memory usage is rather small (in exchange for speed) and pickle does not have to deal with large arrays. But what I really dont understand is, that the error gets thrown when set_parameters is executed, where I would not expect any pickle action to occur.

Each simulation is independent of each other and takes rather long so that I thought: multiprocessing would be a good idea here. However as I am not very expercienced, especially with multiprocessing, I am really unsure what would be an appropriate solution to this problem. (I run Windows with 8 cores and 8GB ram - If any more information is needed I am happy to provide it)

Answer 1

Okay, so it turned out, that the problem was actually not caused by pickle itself but by a bug in an update of the simulator package, which tried to create a numpy array of size (n_timesteps, n_timesteps). Sorry about the confusion.