How to get a secrets "file" into Google Cloud Build so docker compose can read it?

Question

How can I configure Google Cloud Build so that a docker-compose setup can use a secret file the same way as it does when it is run locally on my machine accessing a file?

My Docker-compose based setup uses a secrets entry to expose an API key to a backend component like this (simplified for example):

services:
  backend:
    build: docker_contexts/backend
    secrets:
      - API_KEY
    environment:
      - API_KEY_PATH=/run/secrets/api_key

secrets:
  API_KEY:
    file: ./secrets/api_key.json

From my understanding docker-compose places any files in the secrets section in /run/secrets on the local container for access, that's why the target location is hard-coded to /run/build.

I would like to deploy my docker-compose setup on Google Cloud Build to use this configuration, but the only examples I've seen in documentation have been to load the secret as an environment variable. I have tried to provide this secret to the secret manager and copy it to a local file at /run/secrets like this:

steps:
- name: gcr.io/cloud-builders/gcloud
  # copy to /workspace/secrets so docker-compose can find it
  entrypoint: 'bash'
  args: [ '-c', 'echo $API_KEY > /workspace/secrets/api_key.json' ]
  volumes:
  - name: 'secrets'
    path: /workspace/secrets
  secretEnv: ['API_KEY']
# running docker-compose
- name: 'docker/compose:1.29.2'
  args: ['up', '-d']
  volumes:
  - name: 'secrets'
    path: /workspace/secrets

availableSecrets:
  secretManager:
    - versionName: projects/ID/secrets/API_KEY/versions/1
      env: API_KEY

But when I run the job on google cloud build, I get this error message after everything is built: ERROR: for backend Cannot create container for service backend: invalid mount config for type "bind": bind source path does not exist: /workspace/secrets/api_key.json.

Is there a way I can copy the API_KEY environment variable at the cloudbuild.yaml level so it is accessible to the docker-compose level like it is when I run it on my local filesystem?

Answer 1

If you want to have the value of API_KEY taken from Secret Manager and placed into a text file at /workspace/secrets/api_key.json then change your step to this:

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: "bash"
    args: ["-c", "mkdir -p /workspace/secrets && echo $$API_KEY > /workspace/secrets/api_key.json"]
    secretEnv: ["API_KEY"]

This will:

1. Remove the unnecessary volumes attribute which is not necessary as /workspace is already a volume that persists between steps
2. Make sure the directory exists before you try to put a file in it
3. Use the $$ syntax as described in Use secrets from Secret Manager so that it echoes the actual secret to the file

Note this section:

When specifying the secret in the args field, specify it using the environment variable prefixed with $$.

You can double-check that this is working by adding another step:

  - name: gcr.io/cloud-builders/gcloud
    entrypoint: "bash"
    args: ["-c", "cat /workspace/secrets/api_key.json"]

This should echo out the contents of the file as the build step, allowing you to confirm that:

1. The previous step read the secret
2. The previous step wrote the secret to the file
3. The file was written to a volume that persists across steps

From there you can configure docker-compose to read the contents of that persisted file.