Given a file oneSourceOfTruth.env:
FOO=42
... (many entries)
and a docker-compose.yml:
services:
my-service:
dockefile: ./Dockerfile
env_file: oneSourceOfTruth.env
I'd like to have all variables from oneSourceOfTruth.env available in the Dockerfile during the build step via docker compose build as well as in the container during runtime (docker compose up). The variables are static in the sense that they are once set (before the build) and then never change again for a specific build and the containers that spawn from the build.
Unfortunately, it does not work with env_file option as it only passes the env variables to the containers. The env variables are not available during the build (e.g. with a RUN command inside the Dockerfile).
Sidenote: I don't necessarily need to access the env variables directly inside the docker-compose.yml file itself, e.g. we don't have something like the following in our docker-compose.yml file.
args:
- FOO=$FOO
The oneSourceOfTruth.env file is quite long. For scalability reasons and better code quality, I'd like to regard this file as the "one source of truth" where env variables are declared and set. That is, introducing (removing) an env variable should only imply I have to add (remove) one line in this file alone.
ARG FOO
ENV FOO $FOO
in your Dockerfile. Do this for every single variable. One user knows what I feel about this approach (see here):
This is not a practical solution whatsoever if you have a big environment file. This makes development tedious and unnecessarily annoying where if you can just read environment variables passed from the compose file. Hard-coding variables is never good, because if you would then want to add more environment variables you have to edit multiple files.
The solution proposed here source oneSourceOfTruth.env && docker-compose build did not work for me (env variables were not populated / were empty).
environment section in the docker-compose.yml file as described here and here..env file and env_file optionMaybe this is an XY problem. For the sake of completeness, here is why I'd like to get this problem solved ;)
Inside our Dockerfile, we use the following Rails command to precompile our assets:
RUN DB_ADAPTER=nulldb bundle exec rails assets:precompile
Unfortunately, this will spawn the entire Rails machinery (at least we can avoid connecting to the DB with the DB_ADAPTER=nulldb adapter from here). There existed an option initialize_on_precompile that one could set to false to avoid booting up Rails for this task, see here). However, this option was removed from the Rails codebase, see this commit.
This means we are forced to load env variables also in the precompile task. As stated, with the key env_file in our docker-compose.yml file, the env variables are only available in the containers and not during the build (where we need them for the precompile task). A workaround so far was to use ENV("Foo", nil) everywhere in our ruby on rails code such that the variable default to nil if not specified. This way everything works: during the precompile task, all env variables are nil but we don't need them anyways for the mere task of precompiling our assets. During the production run-time, the env variable will then be available.
But with this approach, we silently ignore the case when we really forget to set an env variable. Then, also during production, the variable will be nil and we only recognize this by the effects it causes. Therefore, our solution so far is really just a workaround and should be fixed. If a variable is not available, it should raise a KeyError during the build step of the project and not just when a user complains that something in our app is not working. This can be achieved by using ENV("Foo") instead of ENV("Foo", nil), but now we get the key error during the precompile task since the env variables are not available in the Dockerfile (where we execute the precompile task). That's exactly the problem ;)
In addition to not matching well with Docker's ENV primitive, the other practical problem with this setup is that the environment gets reset at the end of each RUN command.
If you only need the contents of the file once, then you can use the standard shell . command to read it in within the context of a single RUN command. (Some shells have a similar source command, but it is not part of the shell standard, and I'd avoid it in most cases.)
WORKDIR /app
COPY ./ ./ # includes .env file
RUN . oneSourceOfTruth.env && \
DB_ADAPTER=nulldb bundle exec rails assets:precompile
# RUN echo $Foo # environment variables won't be set any more
# CMD echo $Foo # nor in the eventual container
You can work around the CMD problem by writing a shell script that reads in the environment file and then runs the command it's passed. That script can be the image's ENTRYPOINT.
#!/bin/sh
. /app/oneSourceOfTruth.env
exec "$@"
ENTRYPOINT ["/app/with-environment"]
CMD echo $Foo # will be set now
If you need the environment to be set on every RUN command, then you can override the Dockerfile SHELL to run this script too. This defines the interpreter that's used for shell-syntax RUN and CMD commands (not JSON-array syntax), and you could inject a wrapper script here too.
SHELL ["/app/with-environment", "/bin/sh", "-c"]
RUN echo $Foo
CMD echo $Foo
# CMD ["/bin/sh", "-c", "echo $Foo"] # will not have the environment
The one caution with this last setup is that a docker run or Compose command: override won't see the Dockerfile SHELL and the environment variables won't be set there. You need the ENTRYPOINT wrapper for that.