How to share python libraries from host to multiple docker containers?

Question

We have a setup of multiple docker containers running python microservices. The problem is that a lot of libraries are redundant across the containers. Because of docker isolation, all the libraries are independently installed in each container which is increasing the docker image size.

So, is there a way in which I can share the python libraries across multiple containers by maybe installing all the libraries in the shared folder or any other solution, keeping the code still isolated but allowing to use the shared libraries?

Dockerfile1:

FROM python:3.8.5-slim

RUN apt-get update && \ 
        apt-get install -y \
        git openssh-server 

# Avoid cache purge by adding requirements first
RUN git clone [email protected]:some_org/some_repo_1.git
WORKDIR ./some_repo_1/

RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt

EXPOSE 80

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

Dockerfile2:

FROM python:3.8.5-slim

RUN apt-get update && \ 
        apt-get install -y \
        git openssh-server 

# Avoid cache purge by adding requirements first
RUN git clone [email protected]:some_org/some_repo_2.git
WORKDIR ./some_repo_2/

RUN pip3 install --upgrade pip
RUN pip3 install -r requirements.txt

EXPOSE 81

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "81"]

docker-compose.yml

version: "3.9"
services:
        service_1:
                build:
                        context: './some_repo_1/'
                ports:
                        - '80:80'

        service_2:
                build:
                        context: './some_repo_2/'
                ports:
                        - '81:81'

NOTE: we have to deploy a lot of different microservices docker container having somewhat similar dependencies. So docker image size is a major concern.

PS: I am using docker-compose to run the multiple docker containers.

Answer 1

Say you can split your requirements into a "common" set, and some additional per-application dependencies. In a non-Docker environment, you might run:

python3 -m venv venv1
./venv1/bin/pip install ./requirements-common.txt
./venv1/bin/pip install ./some_repo_1/requirements.txt
./venv1/bin/uvicorn ...
# and similarly for venv2, some_repo_2

You can build your own intermediate Docker image that only contains the common libraries:

FROM python:3.8.5-slim
WORKDIR /app
COPY requirements-common.txt .
RUN pip install -r requirements-common.txt

docker build -t my/app-base .

Then when you build your per-application images, you can depend on that as a base image and have that library set preinstalled. The way the Docker layering works, the underlying library set will be shared across the various derived images.

FROM my/app-base
# Inherits WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 81
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "81"]

Compose doesn't have any specific support for this pattern. If the intermediate image consists of just a set of preinstalled libraries that changes rarely, it might be easiest to manually docker build it as needed, and use the docker-compose.yml you have in the question as-is.