GCP Storage SignedUrl access denied in production

Question

I have next.js app that uses the Google Cloud Storage node.js SDK to upload videos and view them by generating signedUrls. Everything works when running locally, but as soon as I deploy the app (Cloud Run), none of my bucket actions work, particularly generating signedUrls. I generate singedUrls on the backend for the client to view videos I have on the bucket.

When I go to the generated link, I see this:

<Code>SignatureDoesNotMatch</Code>
<Message>Access denied.</Message>
<Details>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Details>

Things I have tried:

• taken the entire service account key and encoded into a base64 and used as a env var. This works fine locally.
• triple checked the env value of the secret on GCP and my local value.
• updated the SDK to latest (7.11.0)
• Given the service account that's running the service "Service Account Token Creator" role.
• adjusted bucket CORS: tried both "*" and "my-domain". Regardless of what the cors is, seems to just work fine locally.
• tried changing content type on the generating url function, but that only returned new errors.
• tried to remove the env injection when making the new Storage() instance and let GCP input its own credentials, but still getting the same error.
• tried to impersonate the service account with ADC locally and that works fine.
• tried to set up a minimal express server with a new service account. Same issue: works locally, but singature mismatch when deployed.

My code:

import { Storage } from "@google-cloud/storage";
import { env } from "process";

const rawKey = env.GCP_BUCKET_HANDLER_KEY ?? "";

const creds = rawKey
    ? JSON.parse(Buffer.from(rawKey, "base64").toString())
    : {};

const storage = new Storage({
    projectId: creds.project_id,
    credentials: creds,
});

const PRIMARY_BUCKET_NAME = env.GCP_PRIMARY_BUCKET_NAME ?? "invalid";

export const primaryBucket = storage.bucket(PRIMARY_BUCKET_NAME);

export async function gcGenerateReadSignedUrl({
    fileName,
    id,
}: GcVideoFilePathProps) {
    const filePath = `video/${id}/${fileName}`;
    const options: GetSignedUrlConfig = {
        version: "v4",
        action: "read",
        expires: Date.now() + 15 * 60 * 1000, // 15 minutes
    };
    const [url] = await primaryBucket.file(filePath).getSignedUrl(options);
    return url;
}

EDIT: Setting up a Go server to generate the signed Url appears to work in deployment. Not sure what's gone wrong: the service account (new one) is the same as that of my Next.js. One alternative I'm strongly leaning towards is moving all my bucket business over to the Go server and query it from the Next server to fetch urls etc.

EDIT2: The fix

My Go server was having issues when I next deployed with injecting my env vars and that's when I discovered there was a trailing whitespace in the bucket name env. I also discovered that I was using signedPostPolicyV4 with formData for the upload, which corrupted the file (strange the file wasn't corrupted before). So now, I:

1. Use the new Storage() and let the library fetch the credentials in the Cloud Run instance and while running locally I set the ADC to point to a key.
2. Fixed the env var with the trailing whitespace. Likely main culprit
3. Fixed the upload to use the correct method.
4. Other fixes to the upload flow on the frontend.

// backend nextjs
const options = {
    version: "v4",
    action: "write",
    expires: Date.now() + 15 * 60 * 1000, // 15 minutes
} satisfies GetSignedUrlConfig;
const [url] = await primaryBucket.file(filePath).getSignedUrl(options);

// frontend nextjs
const upload = await fetch(url, {
    method: "PUT",
    body: file,
});

Now the initial Nextjs solution works smoothly in production. Seems there were many minor points of failure, but when added together made it difficult to pinpoint exactly what was going wrong when. It helped to set up separate test servers to narrow down the issue (shout out to DazWilkin for his suggestions!)

Answer 1

I deployed Google's sample with minor tweaks (for configuration):

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

const BUCKET = process.env.BUCKET;
const OBJECT = process.env.OBJECT;

async function generateV4ReadSignedUrl() {
  const options = {
    version: 'v4',
    action: 'read',
    expires: Date.now() + 15 * 60 * 1000,
  };

  const [url] = await storage
    .bucket(BUCKET)
    .file(OBJECT)
    .getSignedUrl(options);

  console.log('Generated GET signed URL:');
  console.log(url);
  console.log('You can use this URL with any user agent, for example:');
  console.log(`curl '${url}'`);
}

generateV4ReadSignedUrl().catch(console.error);

I deployed the code as a Cloud Run job to avoid messing around with a web server:

BILLING="..."
PROJECT="..."
REGION="..."

BUCKET="gs://${PROJECT}"
OBJECT="..."

ACCOUNT="tester"
EMAIL=${ACCOUNT}@${PROJECT}.iam.gserviceaccount.com

JOB="..."

SERVICES=(
    "artifactregistry"
    "cloudbuild"
    "run"
)
ROLEs=(
    "iam.serviceAccountTokenCreator"
    "storage.objectViewer"
)

gcloud projects create ${PROJECT}

gcloud billing projects link ${PROJECT} \
--billing-account=${BILLING}

for SERVICE in "${SERVICES[@]}"
do
  gcloud services enable ${SERVICE}.googleapis.com \
  --project=${PROJECT}
done

gcloud storage buckets create ${BUCKET} \
--project=${PROJECT}

gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}

for ROLE in "${ROLES[@]}"
do
  gcloud projects add-iam-policy-binding ${PROJECT} \
  --member=serviceAccount:${EMAIL} \
  --role="roles/${ROLE}"
done

# Only to test locally
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL} \
--project=${PROJECT}

# Runs as ACCOUNT
gcloud run jobs deploy ${JOB} \
--source=${PWD} \
--set-env-vars=BUCKET=${BUCKET},OBJECT=${OBJECT} \
--service-account=${EMAIL} \
--region=${REGION} \
--project=${PROJECT}

# Create GCS Object
echo "Hello Freddie" > ${OBJECT}
gcloud storage cp \
  ${OBJECT} \
  ${BUCKET}/${OBJECT} \
--project=${PROJECT}

# Invoke Job | Create Signed URL
ID=$(\
  gcloud run jobs execute ${JOB} \
  --region=${REGION} \
  --project=${PROJECT} \
  --format="value(metadata.name)")

Give it time to complete then:

# Get the "curl" command from the logs
FILTER="labels.\"run.googleapis.com/execution_name\"=\"${ID}\""

gcloud logging read  "${FILTER}" \
--project=${PROJECT} \
--format="value(textPayload)" \
| grep ^curl

Invoke the return curl ... command:

Hello Freddie