Configuring Concourse CI to use AWS Secrets Manager

Question

I have been trying to figure out how to configure the docker version of Concourse (https://github.com/concourse/concourse-docker) to use the AWS Secrets Manager and I added the following environment variables into the docker-compose file but from the logs it doesn't look like it ever reaches out to AWS to fetch the creds. Am I missing something or should this automatically happen when adding these environment variables under environment in the docker-compose file? Here are the docs I have been looking at https://concourse-ci.org/aws-asm-credential-manager.html

version: '3'

services:
  concourse-db:
    image: postgres
    environment:
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: concourse_pass
      POSTGRES_USER: concourse_user
      PGDATA: /database

  concourse:
    image: concourse/concourse
    command: quickstart
    privileged: true
    depends_on: [concourse-db]
    ports: ["9090:8080"]
    environment:
      CONCOURSE_POSTGRES_HOST: concourse-db
      CONCOURSE_POSTGRES_USER: concourse_user
      CONCOURSE_POSTGRES_PASSWORD: concourse_pass
      CONCOURSE_POSTGRES_DATABASE: concourse
      CONCOURSE_EXTERNAL_URL: http://XXX.XXX.XXX.XXX:9090
      CONCOURSE_ADD_LOCAL_USER: test: test
      CONCOURSE_MAIN_TEAM_LOCAL_USER: test
      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
      CONCOURSE_AWS_SECRETSMANAGER_REGION: us-east-1
      CONCOURSE_AWS_SECRETSMANAGER_ACCESS_KEY: <XXXX>
      CONCOURSE_AWS_SECRETSMANAGER_SECRET_KEY: <XXXX>
      CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
      CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}

pipeline.yml example:

jobs:
  - name: build-ui
    plan:
      - get: web-ui
        trigger: true
      - get: resource-ui
      - task: build-task
        file: web-ui/ci/build/task.yml
      - put: resource-ui
        params:
          repository: updated-ui
          force: true
      - task: e2e-task
        file: web-ui/ci/e2e/task.yml
        params:
          UI_USERNAME: ((ui-username))
          UI_PASSWORD: ((ui-password))

resources:
  - name: cf
    type: cf-cli-resource
    source:
      api: https://api.run.pivotal.io
      username: ((cf-username))
      password: ((cf-password))
      org: Blah

  - name: web-ui
    type: git
    source:
      uri: [email protected]:blah/blah.git
      branch: master
      private_key: ((git-private-key))

Answer 1

When storing parameters for concourse pipelines in AWS Secrets Manager, it must follow this syntax,

/concourse/TEAM_NAME/PIPELINE_NAME/PARAMETER_NAME`

If you have common parameters that are used across the team in multiple pipelines, use this syntax to avoid creating redundant parameters in secrets manager

/concourse/TEAM_NAME/PARAMETER_NAME

The highest level that is supported is concourse team level.

Global parameters are not possible. Thus these variables in your compose environment will not be supported.

CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}

Unless you want to change the prefix /concourse, these parameters shall be left to their defaults.

And, when retrieving these parameters in the pipeline, no changes required in the template. Just pass the PARAMETER_NAME, concourse will handle the lookup in secrets manager as per the team and pipeline name.

...
        params:
          UI_USERNAME: ((ui-username))
          UI_PASSWORD: ((ui-password))
...