for a wrapper around perl i need nonblocking subprocesses in python (there is shell io of various types). Additionally i am interested in shell output and return value. Sometimes the return value is 0 but the code did not actually do anything anyways.
So i can do either now with subprocess.call() (nonblocking but not shell output) or subprocess.Popen() (blocking but shell output).
I did some reading but the only solution looks like having a separate queue to do this. Anything easier i missed out?
subprocess.Popen isn't inherently blocking. You can still use proc.stdin.write() and proc.stdout.read(); the only problem is that you risk blocking on one side, or even deadlocking[1], if a pipe fills up. If you know your subprocess will only ever read or write a small amount of data, you don't have to worry about that.
So you can do:
proc = subprocess.Popen(['perl', 'somescript.pl'], stdout=subprocess.PIPE)
buf = StringIO()
CHUNKSIZE = 1024 # how much to read at a time
while True:
# do whatever other time-consuming work you want here, including monitoring
# other processes...
# this keeps the pipe from filling up
buf.write(proc.stdout.read(CHUNKSIZE))
proc.poll()
if proc.returncode is not None:
# process has finished running
buf.write(proc.stdout.read())
print "return code is", proc.returncode
print "output is", buf.getvalue()
break
In a larger app, you could schedule this to happen in your event loop, reactor, etc.
[1] The OS will only allow so much data to fit in a pipe at a time. Say you run cat as your subprocess, and write a ton of data to its stdin. cat will write that data to its own stdout until it fills up, and then it'll block until your program reads some data from stdout and empties the pipe. But your program is still writing to stdin, and cat is no longer reading from it, so that pipe will fill up too. Both processes will be stuck with blocking writes, waiting for the other to read, which will never happen.