Writing Standard Output on Streamlit app in real time

Question

I am trying to display the standard output of a Python script that I am calling with my Streamlit in real time. I have used subprocess Popen to call my python script, however there is a delay after it reaches the subprocess.popen() line and the script starts running after almost 30-40 seconds. I put a print statement on top of the script to check this. Is there any reason behind the lag?

Streamlit app:

import streamlit as st
import subprocess

start_button = st.button("Run External Script")

if start_button:
    command = ["python", script.py]
    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    
    while process.poll() is None:
        line = process.stdout.readline()
        if not line:
           continue
        st.write(line.strip())

script.py:

import time

print("Start")
time.sleep(1)
print("Working")
time.sleep(1)
print("Stop")

I expect the print statements to show up as soon as subprocess.Popen() runs but there is a delay and all the statements are displayed at once without the time.sleep in between.

If I use subprocess.call(), I can see the outputs generate in real time without delay in VSCode Terminal. Is there a way to write in the Streamlit app in real time without delay.

Answer 1

The problem is that python stdout is buffered, this is why you can read process.stdout only after process has run. There are a few ways to disable buffering, the simplest is probably to add -u as argument:

import streamlit as st
import subprocess

start_button = st.button("Run External Script")

if start_button:

    command = ["python", '-u', 'script.py']
    process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)

    while process.poll() is None:
        line = process.stdout.readline()
        if not line:
            continue
        st.write(line.strip())