How to retrieve attributes from selected datum in Altair?

I have a Streamlit dashboard which lets me interactively explore a t-SNE embedding using an Altair plot. I am trying to figure out how to access the metadata of the selected datum so that I can visualize the corresponding image. In other words, given:

selector = alt.selection_single()
chart = (
    alt.Chart(df)
    .mark_circle()
    .encode(x="tSNE_dim1", y="tSNE_dim2", color="predicted class", tooltip=["image url", "predicted class"])
    .add_selection(selector)
)

...is there something akin to

selected_metadata = selector.tooltip
update_dashboard_img(img=selected_metadata["image url"], caption=selected_metadata["predicted class"])

I am aware about image marks but the images are on S3 and there are too many of them to fit in the plot.

Solution

I hate to disagree with the creator of Altair, but I was able to achieve this using streamlit-vega-lite package. This works by wrapping the call to the chart creation function with altair_component():

    from streamlit_vega_lite import altair_component
    ...

    event_dict = altair_component(altair_chart=create_tsne_chart(tsne_df))
    # note: works with selector = alt.selection_interval(), not selection_single()
    dim1_bounds, dim2_bounds = event_dict.get("dim1"), event_dict.get("dim2")
    if dim1_bounds:
        (dim1_min, dim1_max), (dim2_min, dim2_max) = dim1_bounds, dim2_bounds
        selected_images = tsne_df[
            (tsne_df.dim1 >= dim1_min)
            & (tsne_df.dim1 <= dim1_max)
            & (tsne_df.dim2 >= dim2_min)
            & (tsne_df.dim2 <= dim2_max)
        ]
        st.write("Selected Images")
        st.write(selected_images)

        if len(selected_images) > 0:
            for _index, row in selected_images.iterrows():
                img = get_img(row["image url"])
                st.image(img, caption=f"{row['image url']} {row['predicted class']}", use_column_width=True)

The event_dict only contains information about the selector bounds. So, you have to use those values to reselect the data that was selected in the interactive chart.

Note that this package is a POC and has various limitations. Please upvote the Streamlit feature request created by the author of streamlit_vega_lite.