How to use .NET Core secrets in .sh file that is called from docker-compose

Question

Background

I am writing a .NET 5 application and using .net user secrets for my secret keys (database connections & passwords).

Recently I decided to learn Dockers and update my application to work with it so that using Visual Studio I generated a docker file for my API project and then created a docker-compose file that includes the API project & database (and some more irrelevant things for this question).

Almost everything works well. Technically, I can hard-code the secrets, and then the application will work well.

I have some secrets and most of them work fine, e.g: the database connection secrets works well, in the C# code I do the following code and it gets the value from the .net user-secrets:

config.GetConnectionString("Default");

Code Details

I have a secret key that contains a SQL password for the sa user.

dotnet user-secrets set "SA_PASSWORD" "<MySecretPassword>"

Then I have the docker-compose file which is of Linux system and this is part of the code:

sql_in_dc:
  build:
    context: .
    dockerfile: items/sql/sql.Dockerfile
  restart: always
  ports:
    - "1440:1433"
  environment:
    - ACCEPT_EULA=Y
    - SA_PASSWORD=$SA_PASSWORD
    - ASPNETCORE_ENVIRONMENT=Development
    - USER_SECRETS_ID=80a155b1-fb7a-44de-8788-4f5759c60ff6
  volumes:
    - $APPDATA/Microsoft/UserSecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID
    - $HOME/.microsoft/usersecrets/$USER_SECRETS_ID:/root/.microsoft/usersecrets/$USER_SECRETS_ID

As you can see it calls the sql.Dockerfile which is:

FROM mcr.microsoft.com/mssql/server 

ARG PROJECT_DIR=/tmp/devdatabase
RUN mkdir -p $PROJECT_DIR
WORKDIR $PROJECT_DIR
COPY items/sql/InitializeDatabase.sql ./
COPY items/sql/wait-for-it.sh ./
COPY items/sql/entrypoint.sh ./
COPY items/sql/setup.sh ./

CMD ["/bin/bash", "entrypoint.sh"]

Then the setup.sh is:

# Wait for SQL Server to be started and then run the sql script
./wait-for-it.sh sql_in_dc:1433 --timeout=0 --strict -- sleep 5s && \
/opt/mssql-tools/bin/sqlcmd -S localhost -i InitializeDatabase.sql -U sa -P "$SA_PASSWORD"

The Problem

The file setup.sh doesn't recognizes the $SA_PASSWORD environment variable when it comes from the secrets file.

It works well if I change the docker-compose.yml file to:

- SA_PASSWORD=SomePassword

Notes

• I searched for an answer in Google and tried many things but couldn't find exactly my case.
• I know it is possible to use Docker Swarm for the secrets but for now I want to do it without it. I am still learning and prefer that the code will work good and the next step will be to use Docker Swarm / Kubernetes / etc...
• I would be happy to know if there is a fast solution even if it is not the ideal one. Later I will improve it and use better techniques.
• I wrote the code I think that should be enough for the case and the relevant code, but if you need any more data, let me know and I will add it.
• I have it in GitHub in a public repository in a pushed branch. If you want I can share with you the code.

Really big thanks in advance!

Answer 1

The docker-compose.yml is executed on your host OS (so it can use OS environment variables or vars from .env file, or from compose file, ...). The running image - container has it's own set of env variables, in your case that means the running container has no SA_PASSWORD variable. Your usecase would work if you had set the SA_PASSWORD Variable on your host OS.

You can check which variables are set in your container with (if your image comes with bash):

docker exec -it [container id] bash
printenv 

The dotnet-secrets environment variables are created implicit during execution/runtime from Visual Studio (see entry in project file).

So as you mentioned "the compose file can't recognize dotnet-secrets".

You can use:

1. *.env File
2. pass it to compose command: with -e
3. Plain text in compose yml: - SA_PASSWORD=thepassword
4. as host OS Variable

Keep in mind that Visual Studio adds some magic when running or debugging your docker container. See Visual Studio container volume mapping: For ASP.NET core web apps, there might be two additional folders for the SSL certificate and the user secrets, which is explained in more detail in the next section.