Python input() blocks subprocesses from executing

Question

I have a Python script that accepts user input. Different user inputs trigger different functionality. The functionality in question here is one that spawns multiple processes. Here is the script, main.py.

import time
import threading
import concurrent.futures as cf


def executeparallelprocesses():

    numprocesses = 2
    durationseconds = 10

    futures = []

    print('Submit jobs as new processes.')
    with cf.ProcessPoolExecutor(max_workers=numprocesses) as executor:
        for i in range(numprocesses):
            futures.append(executor.submit(workcpu, 500, durationseconds))
            print('job submitted')
        print('all jobs submitted')

        print('Wait for jobs to complete.', flush=True)
        for future in cf.as_completed(futures):
            future.result()

        print('All jobs done.', flush=True)


def workcpu(x, durationseconds):
    print('Job executing in new process.')
    start = time.time()
    while time.time() - start < durationseconds:
        x * x


def main():

    while True:
        cmd = input('Press ENTER\n')
        if cmd == 'q':
            break
        thread = threading.Thread(target=executeparallelprocesses)
        thread.start()

    time.sleep(15)


if __name__ == '__main__':
    main()

When this script is invoked from the terminal, it works as expected (i.e., the subprocesses execute). Specifically, notice the two lines "Job executing in new process." in the example run that follows:

(terminal prompt $) python3 main.py
Press ENTER

Submit jobs as new processes.
Press ENTER
job submitted
job submitted
all jobs submitted
Wait for jobs to complete.
Job executing in new process.
Job executing in new process.
All jobs done.
q
(terminal prompt $)

THE PROBLEM: When the script is invoked from another program, the subprocesses are not executed. Here is the driver script, driver.py:

import time
import subprocess
from subprocess import PIPE


args = ['python3', 'main.py']

p = subprocess.Popen(args, bufsize=0, stdin=PIPE, universal_newlines=True)

time.sleep(1)

print('', file=p.stdin, flush=True)

time.sleep(1)

print('q', file=p.stdin, flush=True)

time.sleep(20)

Notice how "Job executing in new process." is not present in the output from the example run that follows:

(terminal prompt $) python3 driver.py
Press ENTER
Submit jobs as new processes.
Press ENTER
job submitted
job submitted
all jobs submitted
Wait for jobs to complete.
(terminal prompt $)

It seems like the cmd = input('Press ENTER\n') statement in main.py is blocking and preventing the subprocesses from executing. Strangely, commenting out the second time.sleep(1) statement in driver.py causes the main.py subprocesses to spawn as expected. Another way to make this "work" is to add time.sleep(1) inside the loop of main.py, right after thread.start().

This time-sensitive code is brittle. Is there a robust way to do this?

Answer 1

I tried ShadowRanger's suggestion to add a call to multiprocessing.set_start_method():

if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')
    main()

This solved the problem for me. I will read the documentation to learn more about this.