I have the below code that I am running
try:
child = pexpect.spawn(
('some command --path {0} somethingmore --args {1}').format(
<iterator-output>,something),
timeout=300)
child.logfile = open(file_name,'w')
child.expect('x*')
child.sendline(something)
child.expect('E*')
child.sendline(something))
#child.read()
child.interact()
time.sleep(15)
print child.status
except Exception as e:
print "Exception in child process"
print str(e)
Now, the command in pexpect creates subprocess by taking the one of the input from a loop, now everytime it spins up a subprocess I try to capture the logs via the child.read, in this case it waits for that subprocess to complete before going to the loop again, how do I make it to keep running it in the background(I get the logs of command input/output that I enter dynamically, but not of the process that runs thereafter unless I use the read or interact? I used this How do I make a command to run in background using pexpect.spawn? but it uses interact which again waits for that subprocess to complete .. since the loop will be iterated alomst more than 100 times I cannot wait on one to complete before moving to other, as the command in pexpect is an AWS lambda call, all I need to make sure is the command is triggered but I am not able to capture the process output of that call without waiting for it to complete.... Please let me know your suggestions
If you don't actually want to interact with lots of processes in parallel, but instead want to interact with each process briefly, then just ignore it while it runs and move onto interacting with the next one…
# Do everything up to the final `interact`. After that, the child
# won't be writing to us anymore, but it will still be running for
# many seconds. So, return the child object so we can deal with it
# later, after we've started up all the other children.
def start_command(path, arg):
try:
child = pexpect.spawn(('some command --path {0} somethingmore --args {1}').format(path, arg), timeout=300)
child.logfile = open(file_name,'w')
child.expect('x*')
child.sendline(something)
child.expect('E*')
child.sendline(something))
# child.read()
child.interact()
return child
except Exception as e:
print "Exception in child process"
print str(e)
# First, start up all the children and do the initial interaction
# with each one.
children = []
for path, args in some_iterable:
children.append(start_command(path, args))
# Now we just need to wait until they're all done. This will get
# them in as-launched order, rather than as-completed, but that
# seems like it should be fine for your use case.
for child in children:
try:
child.wait()
print child.status
except Exception as e:
print "Exception in child process"
print str(e)
A few things:
Notice from the code comments that I'm assuming the child isn't writing anything to us (and waiting for us to read it) after the initial interaction. If that's not true, things are a bit more complicated.
If you want to not only do this, but also spin up 8 children at a time, or even all of them at once, you can (as shown in my other answer) use an executor or just a mess of threads for the initial start_command calls, and have those tasks/threads return the child object to be waited on later. For example, with the Executor version, each future's result() will be a pexpect child process. However, you definitely need to read the pexpect docs on threads in that case—with some versions of linux, passing child-process objects between threads can break the objects.
Finally, since you will now be seeing things much more out-of-order than the original version, you might want to change your print statements to show which child you're printing for (which also probably means changing children from a list of children to a list of (child, path, arg) tuples or the like).