Running multiple external programs

Question

So, I set up a short script which executes an external program (written in Fortran 77). I want to run multiple instances of the program and since I have 8 cores on my computer the simplest solution I found was:

import subprocess


import os


i = n

while(i<n):
  dire = "dir/Run"+str(i)+"/"
  os.chdir(dire)
  p1 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+1)+"/"
  os.chdir(dire)
  p2 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+2)+"/"
  os.chdir(dire)
  p3 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+3)+"/"
  os.chdir(dire)
  p4 = subprocess.Popen(['./mej'])  
  dire = "dir/Run"+str(i+4)+"/"
  os.chdir(dire)
  p5 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+5)+"/"
  os.chdir(dire)
  p6 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+6)+"/"
  os.chdir(dire)
  p7 = subprocess.Popen(['./mej'])
  dire = "dir/Run"+str(i+7)+"/"
  os.chdir(dire)
  p8 = subprocess.Popen(['./mej'])
  dire = "/Run"+str(i+8)+"/"
  os.chdir(dire)
  p3 = subprocess.Popen(['./mej'])
  exit_codes = [p.wait() for p in p1, p2, p3, p4, p5, p6, p7, p8]
  i = i + 8



print "Job's done!"

Now this worked mostly fine at first, however I just changed to a variable time step and in doing so the time each integration runs for varies significantly. Now the problem is that the script will wait for the slowest one to finish before starting up a new set of integrations. How can I write it so that I will always have 8 instances running?

Answer 1

You could use a thread pool, to keep all CPUs busy:

#!/usr/bin/env python
import os
import subprocess
from multiprocessing.pool import ThreadPool

def run(i):
    working_dir = "dir/Run/" + str(i + 1)
    return i, subprocess.call(os.path.join(working_dir, 'mej'), cwd=working_dir)

results = ThreadPool().map(run, range(n))

As soon as one mej process finishes, the next one is started. No more than os.cpu_count() concurrent worker processes are running at a time.