I have several projects that use very large Python packages. I would like to put each project in a different docker container. To save disk space, I would like to install the large Python packages on the underlying server, so that each docker container can use them without having to install them in each container separately. Each project may have, in addition, some other Python packages, that may be installed locally on the project's container.
Is it possible to have a docker container use some python libraries from the underlying server, and others from the container itself?
The intended way to share files between containers and the host is via volumes or bind-mounts. It seems convinient to mount the python libraries into the container.
To avoid additional issues with the containers when the host's libraries might change, I recommend to setup a virtual environment on the host and mount the resulting directory.
Also, since the libraries will be shared across multiple containers and additional, individual libraries should be installed, we want to make sure that no changes are being reflected to the host, i.e. mount the shared libraries readonly.
Make sure to use the same python version on the host (read: in the virtual environment) and in the containers. In the following minimal example, the host runs python 3.8, so the same version was chosen for the container.
Example:
Create a virtual environment, cd into its root directory and activate it.
python3 -m venv sharedwithdocker
cd sharedwithdocker
source bin/activate
Install a package to be shared with containers, then leave the virtual environment.
pip install requests
deactivate
Create a container that can access our venv. The entire venv folder sharedwithdocker is mounted to /from-host. Note the ro (readonly) which ensures that no changes from within the container are written to the host. This implies that pip will install new packages to the user directory. (This might be not relevant if you install your packages when building the image.)
The environment variables tell python and pip where to look for libraries. Depending on the base image you choose, the default paths might slightly differ. Note: This assumes you are still in the venv directory ($(pwd)).
docker run -it --rm \
--mount type=bind,src=$(pwd),dst=/from-host,ro \
-e PYTHONHOME=/from-host \
-e PYTHONPATH=/usr/local/lib/python3.8:/usr/local/lib/python3.8/lib-dynload:/from-host/lib/python3.8/site-packages \
python:3.8-slim /bin/bash
Within the container, let's install another package:
pip install pytz
As expected, we will see: Defaulting to user installation because normal site-packages is not writeable.
List installed packages to verify that everything works as expected:
pip list
Package Version
------------------ ---------
certifi 2022.12.7
charset-normalizer 3.1.0
idna 3.4
pip 20.0.2
pkg-resources 0.0.0
pytz 2022.7.1
requests 2.28.2
setuptools 44.0.0
urllib3 1.26.15
And python is also able to work with it:
python -c "import pytz; import requests; print('Imports working :-)')"