Not understanding how docker build --secret is supposed to be used

I can get the example working that Docker provides and just about every blog is just a variation of it: just printing out the secret value in the build output.

Cool... but I need to get it into my app that is looking for it and failing to build because it doesn't exist. That much isn't clear to me and I've been unable to figure it out myself.

This is the project structure:

project-root/
  api/
    docker/
      Dockerfile
      mysecret.txt
    src/
      ...
    manage.py

This is the Dockerfile (the last three lines are probably the most relevant):

# syntax=docker/dockerfile:1
FROM python:3.8-slim as python-base
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    POETRY_HOME="/opt/poetry" \
    POETRY_VIRTUALENVS_IN_PROJECT=true \
    POETRY_NO_INTERACTION=1 \
    PYSETUP_PATH="/opt/pysetup" \
    VENV_PATH="/opt/pysetup/.venv"

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"


FROM python-base as builder-base
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        curl \
        build-essential

ENV POETRY_VERSION=1.1.8
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

WORKDIR $PYSETUP_PATH
COPY ./poetry.lock ./pyproject.toml ./
RUN poetry install --no-dev


FROM python-base as development
ENV FASTAPI_ENV=development

COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

COPY ./docker/docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh

WORKDIR $PYSETUP_PATH
RUN poetry install

WORKDIR /app
COPY . .

EXPOSE 5000
ENTRYPOINT /docker-entrypoint.sh $0 $@
CMD ["python", "src/manage.py", "runserver", "0.0.0.0:5000"]


FROM development AS test
RUN --mount=type=secret,id=mysecret export MYSECRET=$(cat /run/secrets/mysecret)
RUN coverage run --omit='src/manage.py,src/config/*,*/.venv/*,*/*__init__.py,*/tests.py,*/admin.py' src/manage.py test src && coverage report

This is the command:

$ docker build -f docker/Dockerfile --no-cache --secret id=mysecret,src=docker/mysecret.txt --target=test --progress=plain .

The app itself is looking for:

SECRET_KEY = os.environ['MYSECRET']
SECRET_KEY = os.environ['mysecret']

But it always fails at the last step of the Dockerfile saying it can't find the necessary env var, MYSECRET or mysecret, to run the app.

I've tried numerous things to resolve the issue and figure out how to appropriately get the secret into the app:

Adding a export MYSECRET step to hopefully add it as env var: printenv shows nothing in the docker run -it <image> bash.
Finding the value in /run/secrets/mysecret in the docker run -it <image> bash which doesn't exist (as it shouldn't).

It just seems like by the time it gets to the RUN coverage ... the value is no longer available.

So what do I need to do here to get docker build --secret working properly with my Dockerfile?

Solution

Short answer

Could you try to combine 2 RUN coverage with RUN --mount ... ?

I think it will be something like that :

RUN --mount=type=secret,id=mysecret \
  export MYSECRET=$(cat /run/secrets/mysecret) \
  && coverage run --omit='src/manage.py,src/config/*,*/.venv/*,*/*__init__.py,*/tests.py,*/admin.py' src/manage.py test src \
  && coverage report

Detail

This --secret flag persists only in one docker layer.

The final image built will not have the secret file

Documentation :

https://docs.docker.com/engine/reference/builder/#run---mounttypesecret

Blog about it:

https://pythonspeed.com/articles/docker-buildkit/

each RUN creates one intermediate layer.

https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

So, according to my understanding,

RUN --mount=type=secret,id=mysecret export MYSECRET=$(cat /run/secrets/mysecret)

will not persist both in your final image and in your last RUN.

EDIT 1

RUN is executed by /bin/sh not /bin/bash so you have to instanciate MYSECRET and export it in 2 steps :

/bin/sh -c [your RUN instructions]

RUN --mount=type=secret,id=mysecret \
    MYSECRET=$(cat /run/secrets/mysecret) \
    && export MYSECRET \
    && coverage run --omit='src/manage.py,src/config/*,*/.venv/*,*/*__init__.py,*/tests.py,*/admin.py' src/manage.py test src \
    && coverage report

Here is my working example :

test.py

print MYSECRET value

import os

print(os.environ["MYSECRET"])

mysecret.txt

contains MYSECRET value
```
Hello
```

Dockerfile

based on python:3.9 image

FROM python:3.9

COPY test.py /tmp/test.py

RUN --mount=type=secret,id=mysecret \
  MYSECRET=$(cat /run/secrets/mysecret) \
  && export MYSECRET \
  && python /tmp/test.py

Build output

...
#7 [3/3] RUN --mount=type=secret,id=mysecret     MYSECRET=$(cat /run/secrets/mysecret)     && export MYSECRET     && python /tmp/test.py
#7 sha256:f0134fcb389100e7094a47f437f8630e67da83447daaf617756c1d8432163bae
#7 0.374 Hello
#7 DONE 0.4s
...