Search code examples
rubyiosubprocesspty

Ruby spawn process, capturing STDOUT/STDERR, while behaving as if it were spawned regularly

What I'm trying to achieve:

  • From a Ruby process, spawning a subprocess
  • The subprocess should print as normal back to the terminal. By "normal", I mean the process shouldn't miss out color output, or ignore user input (STDIN).
  • For that subprocess, capturing STDOUT/STDERR (jointly) e.g. into a String variable that can be accessed after the subprocess is dead. Escape characters and all.

Capturing STDOUT/STDERR is possible by passing a different IO pipe, however the subprocess can then detect that it's not in a tty. For example git log will not print characters that influence text color, nor use it's pager.

Using a pty to launch the process essentially "tricks" the subprocess into thinking it's being launched by a user. As far as I can tell, this is exactly what I want, and the result of this essentially ticks all the boxes.

My general tests to test if a solution fits my needs is:

  • Does it run ls -al normally?
  • Does it run vim normally?
  • Does it run irb normally?

The following Ruby code is able to check all the above:

to_execute = "vim"

output = ""
require 'pty'
require 'io/console'

master, slave = PTY.open
slave.raw!

pid = ::Process.spawn(to_execute, :in => STDIN, [:out, :err] => slave)
slave.close
master.winsize = $stdout.winsize
Signal.trap(:WINCH) { master.winsize = $stdout.winsize }
Signal.trap(:SIGINT) { ::Process.kill("INT", pid) }

master.each_char do |char|
  STDOUT.print char
  output.concat(char)
end

::Process.wait(pid)
master.close

This works for the most part but it turns out it's not perfect. For some reason, certain applications seem to fail to switch into a raw state. Even though vim works perfectly fine, it turned out neovim did not. At first I thought it was a bug in neovim but I have since been able to reproduce the problem using the Termion crate for the Rust language.

By setting to raw manually (IO.console.raw!) before executing, applications like neovim behave as expected, but then applications like irb do not.

Oddly spawning another pty in Python, within this pty, allows the application to work as expected (using python -c 'import pty; pty.spawn("/usr/local/bin/nvim")'). This obviously isn't a real solution, but interesting nonetheless.

For my actual question I guess I'm looking towards any help to resolving the weird raw issue or, say if I've completely misunderstood tty/pty, any different direction to where/how I should look at the problem.

Solution

  • [edited: see the bottom for the amended update]

    Figured it out :)

    To really understand the problem I read up a lot on how a PTY works. I don't think I really understood it properly until I drew it out. Basically PTY could be used for a Terminal emulator, and that was the simplest way to think of the data flow for it:

    keyboard -> OS -> terminal -> master pty -> termios -> slave pty -> shell
                                               |
                                               v
 monitor <- OS <- terminal <- master pty <- termios

    (note: this might not be 100% correct, I'm definitely no expert on the subject, just posting it incase it helps anybody else understand it)

    So the important bit in the diagram that I hadn't really realised was that when you type, the only reason you see your input on screen is because it's passed back (left-wards) to the master.

    So first thing's first - this ruby script should first set the tty to raw (IO.console.raw!), it can restore it after execution is finished (IO.console.cooked!). This'll make sure the keyboard inputs aren't printed by this parent Ruby script.

    Second thing is the slave itself should not be raw, so the slave.raw! call is removed. To explain this, I originally added this because it removes extra return carriages from the output: running echo hello results in "hello\r\n". What I missed was that this return carriage is a key instruction to the terminal emulator (whoops).

    Third thing, the process should only be talking to the slave. Passing STDIN felt convenient, but it upsets the flow shown in the diagram.

    This brings up a new problem on how to pass user input through, so I tried this. So we basically pass STDIN to the master:

      input_thread = Thread.new do
    STDIN.each_char do |char|
      master.putc(char) rescue nil
    end
  end

    that kind of worked, but it has its own issues in terms of some interactive processes weren't receiving a key some of the time. Time will tell, but using IO.copy_stream instead appears to solve that issue (and reads much nicer of course).

    input_thread = Thread.new { IO.copy_stream(STDIN, master) }

    update 21st Aug:

    So the above example mostly worked, but for some reason keys like CTRL+c still wouldn't behave correctly. I even looked up other people's approach to see what I could be doing wrong, and effectively it seemed the same approach - as IO.copy_stream(STDIN, master) was successfully sending 3 to the master. None of the following seemed to help at all:

    master.putc 3
master.putc "\x03"
master.putc "\003"

    Before I went and delved into trying to achieve this in a lower level language I tried out 1 more thing - the block syntax. Apparently the block syntax magically fixes this problem.

    To prevent this answer getting a bit too verbose, the following appears to work:

    require 'pty'
require 'io/console'

def run
  output = ""

  IO.console.raw!

  input_thread = nil

  PTY.spawn('bash') do |read, write, pid|
    Signal.trap(:WINCH) { write.winsize = STDOUT.winsize }
    input_thread = Thread.new { IO.copy_stream(STDIN, write) }

    read.each_char do |char|
      STDOUT.print char
      output.concat(char)
    end

    Process.wait(pid)
  end

  input_thread.kill if input_thread

  IO.console.cooked!
end

Bundler.send(:with_env, Bundler.clean_env) do
  run
end