In my python script I want to run some background subprocesses. I want python not to wait for the subprocess to finish, BUT raise an exception if the subprocess fails.
I believe the typical way to run background processes is this:
subprocess.Popen("script.sh")
# Move on doing something else
But this ignores if script.sh exits unsuccessfullly.
I know I can also do this:
process = subprocess.Popen("script.sh")
process.communicate()
if process.returncode:
raise Exception
# Move on doing something else
But this waits for the process to finish, which I don't want.
For context, I expect script.sh to take ~3 seconds to run and to only fail in the worst case scenario (i.e. the failure of script.sh is not part of my program's logic, but indicates some actual, unrecoverable error). 99% of the time the user shouldn't even know that script.sh is running; but in the 1% case, it's better if python simply crashes than continue running and ignore the error.
How can I make this happen?
I came up with two ideas. Which you prefer depends on your main thread implementation.
The first approach is to use poll. This is better suited for implementations that do something else while waiting for the sub-process to finish. This approach is simple, but you will need to periodically check the status of the sub-process in the main thread.
import subprocess
import time
class PollingWorker:
def __init__(self, args):
self.args = args
self.process = subprocess.Popen(args)
@property
def return_code(self):
return self.process.poll()
def run_polling():
worker = PollingWorker("script.sh")
# Move on doing something else
for i in range(5):
print(i)
time.sleep(1)
# But you need to check return_code periodically.
if worker.return_code:
print("ERROR:", worker.args)
return
The second approach is to use threads. This approach allows you to wait for the sub-process to finish without blocking the main thread by executing the sub-process in a sub-thread. Note that this approach eliminates periodic checks, but makes error handling a bit more complicated.
import _thread
import subprocess
import threading
import time
class ThreadWorker:
def __init__(self, args):
self.args = args
self.thread = threading.Thread(target=self._run_in_thread, args=(args,))
self.thread.start()
def _run_in_thread(self, args):
try:
subprocess.run(args, check=True)
except subprocess.CalledProcessError as e:
print("ERROR:", args, e)
# Note that this method is executed in sub thread.
# That is, raising an exception here will not terminate main thread.
# Instead, following raises KeyboardInterrupt in main thread.
_thread.interrupt_main()
def run_thread():
# You need to keep worker instances to avoid garbage collection.
worker = ThreadWorker("script.sh")
# Move on doing something else
for i in range(5):
print(i)
time.sleep(1)
# This case, you have nothing to take care of.