Search code examples
pythonconcurrencypython-multithreadingfork

run multiple instances of python script simultaneously


I am trying to create 86 instances of task.py to run simultaneously.

import sys
import subprocess

for file in range(86):
    subprocess.call([sys.executable,'task.py',str(file)+'in.csv',str(filen)+'out.csv'])

Solution

  • subprocess.call waits for command to complete. Use subprocess.Popen instead:

    import sys
    import subprocess
    
    procs = []
    for i in range(86):
        proc = subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)])
        procs.append(proc)
    
    for proc in procs:
        proc.wait()