Altair tooltip with local PIL image

Question

Following the official example from the docs I can successfully create an altair scatter plot with each point having a tooltip:

import altair as alt
import pandas as pd

source = pd.DataFrame.from_records(
    [{'a': 1, 'b': 1, 'image': 'https://altair-viz.github.io/_static/altair-logo-light.png'},
     {'a': 2, 'b': 2, 'image': 'https://avatars.githubusercontent.com/u/11796929?s=200&v=4'}]
)
alt.Chart(source).mark_circle(size=200).encode(
    x='a',
    y='b',
    tooltip=['image']  # Must be a list for the image to render
)

The result will look as follows

However I want to use a local image which I loaded via PIL. If i just replace the URL-string with the PIL object in the tooltip I get the error: TypeError: Object of type 'Image' is not JSON serializable.

I then tried to convert to image to JSON following this SO answer

My code now looks as follows:

import json
from io import BytesIO
import base64
def image_formatter2(im):
    with BytesIO() as buffer:
        im.save(buffer, 'png')
        data = base64.encodebytes(buffer.getvalue()).decode('utf-8')
    
    return json.dumps(data)
    


temp = some_local_PIL_image.thumbnail((90,90))

source = pd.DataFrame.from_records(
    [{'a': 1, 'b': 1, 'image': image_formatter2(temp)},
     {'a': 2, 'b': 2, 'image': 'https://avatars.githubusercontent.com/u/11796929?s=200&v=4'}]
)
alt.Chart(source).mark_circle(size=200).encode(
    x='a',
    y='b',
    tooltip=['image']  # Must be a list for the image to render
)

My tooltip is now empty! How can I display the local image in the tooltip?

---

Version info

IPython          : 7.16.1
jupyter_client   : 7.0.6
jupyter_core     : 4.8.1
altair           : 4.1.0

Answer 1

Ok I kind of figured it out. One has to add a prefix data:image/png;base64, to the data. The code is now as follows:

import PIL.Image
from io import BytesIO
import base64
import json

def image_formatter2(im):
    with BytesIO() as buffer:
        im.save(buffer, 'png')
        data = base64.encodebytes(buffer.getvalue()).decode('utf-8')
    
    return f"data:image/png;base64,{data}" # <--------- this prefix is crucial
    


temp = some_local_PIL_image.thumbnail((90,90))

source = pd.DataFrame.from_records(
    [{'a': 1, 'b': 1, 'image': image_formatter2(temp)},
     {'a': 2, 'b': 2, 'image': 'https://avatars.githubusercontent.com/u/11796929?s=200&v=4'}]
)
alt.Chart(source).mark_circle(size=200).encode(
    x='a',
    y='b',
    tooltip=['image']  # Must be a list for the image to render
)

Result

---

Since I didn't know the term/syntax and this may help others as well: The term one has to look for is Data URLs. Here is a nice intro: https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs