pexpect run long bash command hang wxpython GUI

Question

I am programming a GUI software has a terminal window (wxCtrl) to display external program's output in real time while it is executing.

I tried subprocess.Popen, this doesn't work as expected because it will hang my GUI while it is running, and only gives the output after the execution finished.

def miExecuteCmd(self, cmd):
    self.terminal.addText("\n###\n\n")
    self.terminal.addText("Executing: %s\n" % cmd)
    args = shlex.split(cmd)
    p = subprocess.Popen(args, stdout = subprocess.PIPE)
    output = p.stdout.readlines()
    output = "".join(output)
    self.terminal.addText(output)
    if (p.returncode != None and p.returncode != 0 ):
        self.terminal.addText("Command Execution Problem, return code is %d\n" % p.returncode)
    return output 

Now I'm trying to use pexpect, I read this post, how to use pexpect to get spontaneous output of subprocess in python

So I coded something like,

def miExecuteCmd(self, cmd):
    self.terminal.addText("\n###\n\n")
    self.terminal.addText("Executing: %s\n" % cmd)
    output = []
    child = pexpect.spawn(cmd)
    while True:
        try:
            child.expect('\n')
            line = child.before
            output.append(line)
            self.terminal.addText(line)
        except pexpect.EOF:
            break 
    if child.exitstatus != None and child.exitstatus != 0:
        line = "Command Execution Problem, return code is %d\n" % child.exitstatus
        self.terminal.addText(line)
        output.append(line)

    output = "".join(output)

    return output

But still the GUI will freeze while I used a long time running cmd.

So I am asking for a simple pexpect solution allowing me to operate my GUI and see the cmd's output at the same time.

I read the pexpect document, it seems pexpect.spawn() should start a separated thread for the command, now I'm confused whether put pexpect.spawn() in a new thread.

Answer 1

Finally I worked out a fine solution for my problem, GUI freeze and threading communication in wxPython.

One should have read this article http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads , it is a combination of threading, wx.CallAfter(), PubSub to solve the thread communication problem. So in my case, just add the pexpect to indicate what to communicate.

Here's my run() example. you will need to see the example in the link above.

def run(self):
        wx.CallAfter(self.sendToMainThread, "\n###\n\n")
        text = "Executing: %s\n" % (self.cmd)
        wx.CallAfter(self.sendToMainThread, text)

        child = pexpect.spawn(self.cmd) 
        while True:
            try:
                if self.stopFlag:
                    line = "Stop Buttont Clicked, Stopping External Command... \n"
                    wx.CallAfter(self.sendToMainThread, line)
                    child.terminate(True)
                    child.close()
                    break
                child.expect('\n')
                line = child.before
                wx.CallAfter(self.sendToMainThread, line)
            except pexpect.EOF:
                child.close()
                break
        if child.exitstatus != None and child.exitstatus != 0:
            line = "Command Execution Problem, return code is %d\n" % child.exitstatus
            wx.CallAfter(self.sendToMainThread, line)
        #it looks like next line never happens because of the exception handling above.
        #child.close() make sure child return a code.
        elif child.exitstatus == None:
            line = "Command Execution was interrupted.\n"
            wx.CallAfter(self.sendToMainThread, line)

        #sending an end signal to main thread. command is finished.
        wx.CallAfter(Publisher().sendMessage, "endthread", True)