I have a bash script which interacts with the user using read prompts, for example:
#!/bin/bash
echo start
read -p "Press something to continue " -n 1 -r
echo got $REPLY.
read -p " Press again " -n 1 -r
echo " done"
I want to run the script as a subprocess using python, while both capturing the output and displaying it in real time to the user. Browsing various threads here, I have come up with this solution that does work for me, however very slowly if the script produces large outputs (significantly longer than running the script manually), probably due to the continuous usage of flush for every character:
import subprocess
with subprocess.Popen(['./a.sh'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as proc:
output_text = ''
while c := proc.stdout.read(1):
print(c, end='', flush=True)
output_text += c
The "read(1)" is needed since read is blocking and trying to read larger chunks means that the prompt message won't be fully displayed before the user is expected to enter input. The flush is also necessary since without it the input prompt message is only printed after the user is required to enter input.
I'm looking for a solution that will allow reading large chunks of data (hence reducing the number of calls for print), while still making sure that the full prompt appears to the user before waiting for their input.
Also, for the purpose of this question, the bash script is given as input and cannot be changed, nor do we have any information about its structure. I'm trying to write code that will generically execute such scripts while both presenting and capturing their output.
Similarly to using larger chunks, readline wouldn't work here as it expects EOL, but EOL is only received after the user enters input for the prompt.
I've tried looking at pexpect, but all the examples I've seen assume they know which prompt to look for, while in this case I assume no knowledge of the script I'm running.
In various similar questions I've seen, the solution either uses a single char read (e.g., by calling read(1) or setting buffer size to 1), or uses readline - non of which seem suitable for my current use case. examples: How to get realtime Python subprocess output? Getting realtime output using subprocess Getting realtime and full output using subprocess
Instead of proc.stdout.read(1) you can from os import read and read(proc.stdout.fileno(), 4096) - this will return also if less than the given length is readable.