Run a Python subprocess in parallel

Question

I'm writing a Python program to automate tasks. My program executes another program, let's call it "program 1", with the command subprocess:

subprocess.call(cmd, cwd=current_working_directory,shell=True,stdout=subprocess.DEVNULL,stderr=subprocess.STDOUT)

The cmd command executes "program 1". But "program 1" never ends (I can't correct this because I don't have access to the source code of "program 1").

I get an output.log file that fills up as "program 1" executes. I've created a Python script that reads this file every 60 seconds and detects when "program 1" has finished. The aim is to ensure that when the Python script detects that "program 1" has finished, it kills it.

I'd like to be able to run this Python script at the same time as "program 1", but I can't.

I've tried two subprocesses in a row:

subprocess.call(cmd, cwd=current_working_directory,shell=True,stdout=subprocess.DEVNULL,stderr=subprocess.STDOUT) # Executes the program 1

 subprocess.call(cmd2, cwd="/home/.../Documents/pythonScriptFolder",shell=True,stdout=subprocess.DEVNULL,stderr=subprocess.STDOUT) # Execute Python script to verify program 1 termination

I tried it with Popen:

procs = [Popen(commands[i],cwd=Lcwd[i],shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE) for i in [0,1]]

This does not work. How can I fix it?

Answer 1

Python's subprocess module enables parallel subprocess execution.

import subprocess

def run_command(command):
    process = subprocess.Popen(command, shell=True)
    process.wait()  # Wait for the process to complete

# List of commands to run in parallel
commands = [
    'command_1',
    'command_2',
    'command_3',
    # Add more commands as needed
]

# Create a list to store the subprocess objects
processes = []

# Start each command in parallel
for command in commands:
    process = subprocess.Popen(command, shell=True)
    processes.append(process)

# Wait for all subprocesses to complete
for process in processes:
    process.wait()

Therun_command function runs a single command usingsubprocess.Popen, creating a separate subprocess object for each command and starting it. It waits for all subprocesses to complete. Modify the commands list to include specific commands or scripts. The shell argument is used, but you can pass individual command- line arguments directly by passing them as a list of arguments. Acclimate the law as demanded for your specific conditions.