Cloud Build - Credentials error using exec wrapper to connect to Cloud SQL Proxy + other GCP resources

Question

I'm attempting to use the exec-wrapper in a Cloud Build step to run the Cloud SQL Proxy and run a Node script to do a custom database migration. Here is what my cloud build config looks like:

steps:
- name: gcr.io/cloud-builders/docker
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/api-stg', '.']
- name: gcr.io/cloud-builders/docker
  args: ['push', 'gcr.io/$PROJECT_ID/api-stg']
- name: gcr.io/cloud-builders/gcloud
  args: ['app', 'deploy', 'app-stg.yaml', '--image-url=gcr.io/$PROJECT_ID/api-stg']
- name: "gcr.io/google-appengine/exec-wrapper"
  args: ["-i", "gcr.io/$PROJECT_ID/api-stg",
         "-s", "$PROJECT_ID:us-central1:<Cloud SQL Instance Name>",
         "--", "scripts/management/custom_migration"]

images: ['gcr.io/$PROJECT_ID/api-stg']
timeout: 1200s # 20 minutes

And in my custom_migration.js file I have things like:

const {Storage} = require('@google-cloud/storage');

const storage = new Storage();
const bucket = storage.bucket(BUCKET_NAME);
const file = await bucket.file(key);
const result = await new Promise(resolve => file.download((err, data) => {...}));
...

This causes the following error from the google-auth-library:

Error: The file at /root/.google/credentials does not exist, or it is not a file. 
Error: ENOENT: no such file or directory, lstat '/root/.google'

My App Engine Flexible environment is able to run this code when deployed in a new version, but the same code in a Cloud Build step isn't credentialed correctly. How can I allow the exec-wrapper to use the default credentials of my App Engine Flexible environment?

Answer 1

This step works for me

steps:
  - name: 'node:14-alpine'
    entrypoint: 'sh'
    args:
      - -c
      - |
        wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
        chmod +x cloud_sql_proxy
        ./cloud_sql_proxy -instances=my-project-id:us-central1:vertx=tcp:5432 &
        npm install @google-cloud/storage pg
        node index-test.js

Because I don't know the content of your custom container, you can try to adapt something like this

steps:
  - name: 'gcr.io/$PROJECT_ID/api-stg'
    entrypoint: 'sh'
    args:
      - -c
      - |
        wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
        chmod +x cloud_sql_proxy
        ./cloud_sql_proxy -instances=my-project-id:us-central1:vertx=tcp:5432 &
        npm install @google-cloud/storage pg
        node scripts/management/custom_migration

---

About the issue with the standard library, my supposition is a conflict between the AppEngine environment and the Cloud Build environment.

If you have a look to the Google Auth library, you can see a case for App Engine credentials, and another one for Compute credentials. The compute is used in standard on any Google Cloud services (Cloud Run, Cloud Functions, Compute Engine, Cloud Build,...) but App Engine has its specificities.