Python multiprocessing.pool to speed up a method in a class without return

Question

I have a class that I use to process a bunch of clusters with interesting signal in my research. I found the methods that take so long time to process and will be necessary to take a more fast way to compute, because the number of clusters will increase.

Following my method to compute contributions in the signal. The line for Etruth in setEclusters: was used before in serial mode, and is commented to run in parallel mode:

class cluster_mxn:
    def ComputeXTmultiCore(self, idx):
            ## Compute the XT inductive and capacitive to build a relative contributions on each cell on the cluster
        XTc, XTl, Noise, clusSampXT_C, clusSampXT_L, clusSampNoise, clusSampCell = [], [], [], [], [], [], []
        nSamp = 4        

        setEclusters = self.clusEtruth
        
        #for Etruth in setEclusters: ## approach before change to parallel mode
        Etruth = setEclusters[idx]
        i, j      = np.shape(Etruth)
        g_tau_0   = genTau_0(nSamp*i*j)
        vectDelay = genSampDelay(nSamp*i*j)

        clusCellSamp, clusXTcSamp, clusXTlSamp  = genCellSamples(vectDelay, g_tau_0), genXTcSamples(vectDelay, g_tau_0), genXTlSamples(vectDelay, g_tau_0)

        clusNoise = genNoise(i*j, norm=True)
        clusNoise = clusNoise.reshape(Etruth.shape)

        XTc_Cluster, XTl_Cluster = self.RelativeClusters(Etruth)

        XTc.append(XTc_Cluster), XTl.append(XTl_Cluster), Noise.append(clusNoise)
                                    
        self.clusXT_C  = XTc
        self.clusXT_L  = XTl
        self.clusNoise = Noise

And here, how I'm calling:

idx = clus2_zee.SetClusterSize()
if __name__ == '__main__':
    with Pool(4) as pool:        
        pool.map((clus2_zee.ComputeXTmultiCore), range(len(idx)) )
        pool.close()
        pool.join() 

The code run in parallel mode, but didn't append the variables XTc, XTl and Noise in each iteration, and didn't share the results in self.xxx.

How can I indicate properly the variables that each process should store results to share in my class after processing?

Answer 1

Multiprocessing isn't a magic bullet. It spawns new copies of the interpreter which are isolated except for the communication primitives provided by multiprocessing (and anything else you might build). Specifically they can't clobber each other's ram. So you need to do the communication explicitly, either by returning a value (or pushing it into a queue) or by setting up shared muteable state. And as the docs note, you almost always don't want to have shared state. In this case there's no reason I can see not to return the vars you need and sort it all out later. Doing so will be faster, too, since you won't need locks and the associated waiting.

See the docs for more detail.