Certain (py3-pandas, py3-scipy) packages installed through apk on python:3.12-alpine are not accessible since they are installed in /usr/lib/python3.11/site-packages inspite of specifying $PYTHONPATH=/usr/local/lib/python3.12/site-packages
For reference, here's the relevant portion of the Dockerfile:
FROM python:3.12-alpine
ENV PYTHONUNBUFFERED=1
ENV PYTHONPATH=/usr/local/lib/python3.12/site-packages
RUN apk add --no-cache \
gcc g++ libffi-dev musl-dev \
py3-pandas py3-scipy \
&& pip3 install pip-tools
WORKDIR /app
COPY requirements.in .
RUN pip-compile requirements.in > requirements.txt \
&& pip3 install -r requirements.txt
ENTRYPOINT ["sh"]
apk hard-codes the binaries it creates; there is no sane way to install these packages from apk and expect them to continue to work on a completely different Python version. If you examine the .apk file itself, you will notice that it contains these paths inside the package; there is nothing you can do anymore at that point to change what's inside the APK file.
As a crude workaround, you can put multiple directories in sys.path or PYTHONPATH; these variables are additive. Demo:
PYTHONPATH=/usr/lib/python3.11/site-packages \
python -c 'import sys; print(sys.path)'
Output:
['', '/usr/lib/python3.11/site-packages', '/usr/local/lib/python312.zip', '/usr/local/lib/python3.12', '/usr/local/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/site-packages']
So, you can speculatively set PYTHONPATH=/usr/lib/python3.11/site-packages and it will use the 3.11 packages in addition to the default locations which are in sys.path by default.
That still doesn't remove the problem that packages compiled for 3.11 might simply not work on 3.12. In fact, if you try, it fails:
/ # PYTHONPATH=/usr/lib/python3.11/site-packages:/usr/local/lib/python3.12/lib-dynload python -c 'import pandas'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.11/site-packages/pandas/__init__.py", line 16, in <module>
raise ImportError(
ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.12 from "/usr/local/bin/python"
* The NumPy version is: "1.25.2"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
(The above also coincidentally demonstrates how to put multiple directories in PYTHONPATH; separate them with a colon.)
So, in summary, you need to compile these packages yourself. If you wanted to, you could do it once and put the results in an APK repository of your own, and download from there instead if you have multiple images or nodes which need to have these packages installed.