Node.js can't read python subprocess stdout when it read from stdin

Question

I have a node.js script which starts a python subprocess and reads its stdout. This works as long as the python process does not try to read from stdin. Then the parent process does not get anything from the child.

I have the node.js script and two python test cases here: (both examples work if you comment the lines that try to read from stdin)

First child:

import sys

print('before')

for line in sys.stdin:
    print(line)

print('after')

Second child:

import sys 

print('before')

while True:
    line = sys.stdin.readline()

    if line != '':
        print(line)
    else:
        break

print('after')

Parent:

const spawn = require('child_process').spawn;

let client = spawn('python', ['test1.py'], {cwd: '/tmp'});

client.stdout.on('data', (data) => {
  console.log(data.toString());
});

client.stderr.on('data', (data) => {
  console.log(data.toString());
});

client.on('close', () => {
  console.log('close');
});

client.on('exit', () => {
  console.log('exit');
});


client.on('disconnect', () => {
  console.log('disconnect');
})

Answer 1

A process stdout can be unbuffered, line buffered or block buffered depending on how the process was started. In particular, programs started from the console are line buffered and programs whose stdout is redirected (to a pipe or file) are block buffered. This is done to increase overall program efficiently. People want to see things right away so terminals are line buffered but other programs and files can wait and get things in bigger blocks, so they are block buffered.

You can fix the problem on the python side by forcing the data to be fluxhed on each write. You can do this with the print statement or with the sys.stdout object itself

print('line 1', flush=True)
print('line 2')
print('line 3')
sys.stdout.flush()

You can also fix it on the node.js side by emulating a terminal, basically tricking the program into thinking it is displaying to a user.

const spawn = require('pty.js').spawn;

This is more generic - you don't need the child's cooperation to make it work. But it can get complicated. Some child processes get information about the attached tty to do more complicated things like create menus or color output. But its frequently a great choice.