Pulling real time data and update in Streamlit and Asyncio

The goal is to pulling real time data in the background (say every 5 seconds) and pull into the dashboard when needed. Here is my code. It kinda works but two issues I am seeing: 1. if I move st.write("TESTING!") to the end, it will never get executed because of the while loop. Is there a way to improve? I can imagine as the dashboard grows, there will be multiple pages/tables etc.. This won't give much flexibility. 2. The return px line in the async function, I am not very comfortable with it because I got it right via trial and error. Sorry for being such a newbie, but if there are better ways to do it, I would really appreciate.

Thank you!

import asyncio
import streamlit as st
import numpy as np

st.set_page_config(layout="wide")

async def data_generator(test):
    while True:
        with test:
            px = np.random.randn(5, 1)
        await asyncio.sleep(1)
        return px

test = st.empty()
st.write("TESTING!")

with test:
    while True:
        px = asyncio.run(data_generator(test))
        st.write(px[0])

Solution

From my experience, the trick to using asyncio is to create your layout ahead of time, using empty widgets where you need to display async info. The async coroutine would take in these empty slots and fill them out. This should help you create a more complex application.

Then the asyncio.run command can become the last streamlit action taken. Any streamlit commands after this wouldn't be processed, as you have observed.

I would also recommend to arrange any input widgets outside of the async function, during the initial layout, and then send in the widget output for processing. Of course you could draw your input widgets inside the function, but the layout then might become tricky.

If you still want to have your input widgets inside your async function, you'd definitely have to put them outside of the while loop, otherwise you would get duplicated widget error. (You might try to overcome this by creating new widgets all the time, but then the input widgets would be "reset" and interaction isn't achieved, let alone possible memory issue.)

Here's a complete example of what I mean:

import asyncio
import pandas as pd
import plotly.express as px

import streamlit as st
from datetime import datetime


CHOICES = [1, 2, 3]


def main():
    print('\nmain...')

    # layout your app beforehand, with st.empty
    # for the widgets that the async function would populate
    graph = st.empty()
    radio = st.radio('Choose', CHOICES, horizontal=True)
    table = st.empty()
    
    try:
        # async run the draw function, sending in all the
        # widgets it needs to use/populate
        asyncio.run(draw_async(radio, graph, table))
    except Exception as e:
        print(f'error...{type(e)}')
        raise
    finally:    
        # some additional code to handle user clicking stop
        print('finally')
        # this doesn't actually get called, I think :(
        table.write('User clicked stop!')
    
    
async def draw_async(choice, graph, table):
    # must send in all the streamlit widgets that
    # this fn would interact with...

    # this could possibly work, but layout is tricky
    # choice2 = st.radio('Choose 2', CHOICES)

    while True:
        # this would not work because you'd be creating duplicated
        # radio widgets
        # choice3 = st.radio('Choose 3', CHOICES)

        timestamp = datetime.now()
        sec = timestamp.second

        graph_df = pd.DataFrame({
            'x': [0, 1, 2],
            'y': [max(CHOICES), choice, choice*sec/60.0],
            'color': ['max', 'current', 'ticking']
        })

        df = pd.DataFrame({
            'choice': CHOICES,
            'current_choice': len(CHOICES)*[choice],
            'time': len(CHOICES)*[timestamp]
        })

        graph.plotly_chart(px.bar(graph_df, x='x', y='y', color='color'))
        table.dataframe(df)

        _ = await asyncio.sleep(1)


if __name__ == '__main__':
    main()