Search code examples
bashdockergoogle-cloud-platformdocker-composegcloud

ERROR: (gcloud.auth.activate-service-account) The .json key file is not in a valid format -- via impersonate-service-account

Is it possible to use short-lived credentials, with docker-compose, to run a bash scripted gcloud command?

Related posts that I attempted to use but they are 5+ years old and I've been led to believe that the gcloud auth command has changed during this time:

Setup

  • there is a lot going on but I've attempted to abbreviate to the relevant parts

Makefile

auth: ## commands for short lived auth
    @gcloud config set project ${GCP_PROJECT}
    @gcloud auth application-default login --impersonate-service-account="inst-dataflow-svc@${GCP_PROJECT}.iam.gserviceaccount.com"
    @gcloud auth configure-docker $(REGION)-docker.pkg.dev

gcloud-flex-build: ## build & push base docker image
    docker-compose build gcloud-build-flex-local
    docker-compose run gcloud-build-flex-local

docker-compose.yaml

version: '3.4'
services:
  gcloud-build-flex-local:
    build:
      dockerfile: docker/gcloud-build-flex-template.dockerfile
      context: .
    image: us-central1-docker.pkg.dev/gcp-project/dataflow-docker-registry/local-build/pubsub-to-gbq-build-flex-template
    volumes:
      - type: bind
        source: ${HOME}/.config/gcloud/
        target: /tmp

docker/gcloud-build-flex-template.dockerfile

FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1

COPY docker/scripts/gcloud-build-flex-template.sh /app/gcloud-build-flex-template.sh
COPY dataflow/pubsub-to-gbq/pubsub-to-gbq-metadata /app/pubsub-to-gbq-metadata

WORKDIR /app

ENTRYPOINT "/app/gcloud-build-flex-template.sh"

/app/gcloud-build-flex-template.sh

#!/bin/bash

set -euo pipefail

SERVICE_ACCOUNT_EMAIL=inst-dataflow-svc@gcp-project.iam.gserviceaccount.com
GCP_PROJECT=gcp-project

export GOOGLE_APPLICATION_CREDENTIALS=/tmp/application_default_credentials.json

# debugging
echo $GOOGLE_APPLICATION_CREDENTIALS
ls -lah /tmp/
cat $GOOGLE_APPLICATION_CREDENTIALS

gcloud auth activate-service-account $SERVICE_ACCOUNT_EMAIL --project=$GCP_PROJECT --key-file=$GOOGLE_APPLICATION_CREDENTIALS

Execution

make auth
make gcloud-flex-build

Error

ERROR: (gcloud.auth.activate-service-account) The .json key file is not in a valid format.

make: *** [gcloud-flex-build] Error 1

stdout (abbreviated)

docker-compose build gcloud-build-flex-local
[+] Building 0.4s (9/9) FINISHED
...
docker-compose run gcloud-build-flex-local

drwxr-xr-x 17 root root  544 Dec 30 10:36 .
drwxr-xr-x  1 root root 4.0K Dec 30 10:40 ..
-rw-------  1 root root  591 Dec 30 10:36 application_default_credentials.json

{
  "delegates": [],
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/[email protected]:generateAccessToken",
  "source_credentials": {
    "client_id": "alphanumeric string .apps.googleusercontent.com",
    "client_secret": "alphanumeric string",
    "refresh_token": "alphanumeric string",
    "type": "authorized_user"
  },
  "type": "impersonated_service_account"
}

I can make it work via docker run by spoofing the credentials to include only the "source_credentials" object, passed in as a volume, but this same trick doesn't seem to work with docker-compose running a script inside a container...

Solution

  • There is a similar type of configuration mentioned in this document. This involves three major steps:

    • Create short-lived credentials for your service account and download your service account keys.
    • Create the configuration files for making your docker environment up. Use the above cred files for granting required permissions.
    • Once you have all the configuration files in place use your docker-compose commands for making your environment up.

    Follow this documentation for more details.