Python subprocess doesn't work without sleep

Question

I'm working on a Python launcher which should execute a few programs in my list by calling subprocess. The code is correct, but it works very strangely.

In short, it doesn't work without some sleep or input command in main.

Here is the example:

import threading
import subprocess
import time

def executeFile(file_path):
  subprocess.call(file_path, shell=True)


def main():
  file = None

  try:
      file = open('./config.ini', 'r');
  except:
    # TODO: add alert widget
    print("cant find a file")

  pathes = [ path.strip() for path in file.readlines() ]

  try:
    for idx in range(len(pathes)):
        print(pathes[idx])
        file_path = pathes[idx];
        newThread = threading.Thread(target=executeFile, args=(file_path,))
        newThread.daemon = True
        newThread.start()
  except:
    print("cant start thread")


  if __name__ == '__main__':
    main()

    # IT WORKS WHEN SLEEP EXISTS
    time.sleep(10)

    # OR
    # input("Press enter to exit ;)")

but without input or sleep it doesn't work:

if __name__ == '__main__':
   # Doesn't work
   main()

Could someone explain me, please, why it happens?

I have some idea but I'm not sure. Maybe it's because subprocess is asynchronyous and the program executes and closes itself BEFORE the subprocess execution.

In case of sleep and input, the program suspends and subprocess has enough time to execute.

Thanks for any help!

Answer 1

As soon as the last thread is started, your main() returns. That in turn will exit your Python program. That stops all your threads.

From the documentation on daemon threads:

Note: Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.

The simple fix would be to not use daemon threads.

---

As an aside, I would suggest some changes to your loop. First, iterate over pathes directly instead of using indices. Second; catch errors for each thread seperately, so one error doesn't leave remaining files unprocessed.

for path in pathes:
    try:
        print(path)
        newThread = threading.Thread(target=executeFile, args=(path,))
        newThread.start()
    except:
        print("cant start thread for", path)

---

Another option would be to skip threads entirely, and just maintain a list of running subprocesses:

import os
import subprocess
import time


def manageprocs(proclist):
    """Check a list of subprocesses for processes that have
       ended and remove them from the list.

    :param proclist: list of Popen objects
    """
    for pr in proclist:
        if pr.poll() is not None:
            proclist.remove(pr)
    # since manageprocs is called from a loop,
    # keep CPU usage down.
    time.sleep(0.5)


def main():

    # Read config file
    try:
        with open('./config.ini', 'r') as f:
            pathes = [path.strip() for path in f.readlines()]
    except FileNotFoundError:
        print("cant find config file")
        exit(1)

    # List of subprocesses
    procs = []
    # Do not launch more processes concurrently than your
    # CPU has cores.  That will only lead to the processes
    # fighting over CPU resources.
    maxprocs = os.cpu_count()
    # Launch all subprocesses.
    for path in pathes:
        while len(procs) == maxprocs:
            manageprocs(procs)
        procs.append(subprocess.Popen(path, shell=True))
    # Wait for all subprocesses to finish.
    while len(procs) > 0:
        manageprocs(procs)


if __name__ == '__main__':
    main()