Search code examples
python-multiprocessing

Python multiprocessing.pool sequential run of processes

I am pretty new to python and the 'multiprocessing' module in particular. However, I have managed to write a very simple script to run multiple processes (say 100) on 24 cpus. However, I have noticed that the process are not run sequentially but instead randomly. Is there a way for the processes to be run sequentially. Here is my code:

#!/usr/bin/env python

import multiprocessing
import subprocess


def prcss(cmd):
    sbc = subprocess.call
    com = sbc(cmd, shell='True')
    return (com)


if __name__=='__main__':

    cmd = []
    for j in range(1,11):
        for i in range(10):
            sis = '~/codes-paul/sisyphus/sisyphus '+str(j)+'/sisyphus.setup > '+str(j)+'/out'+str(i)+'.dat'
            cmd.append(sis)

    pool=multiprocessing.Pool(processes=24)
    pool.map(prcss,cmd)

After I have run the python code, I do 'ps -ef | grep myname'. Instead of getting:

'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat'
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out1.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out2.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat
.
.
.
.

I am getting:

'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat'
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out6.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out9.dat.dat 
.
.
.
.

Any idea why the commands are not run sequentially?

Solution

  • Since you are creating a pool of processes, the commands are actually started sequentially, but you have no guarantee on which process is going to finish first. You will notice that the order will vary almost every time you run your code.