After creating a new Google Cloud project and following all steps carefully from this documentation to create & deploy a new GAE app, gcloud app deploy yields the following error:
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 12 files to Google Cloud Storage ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build:
com.google.net.rpc3.client.RpcClientException: <eye3 title='/ArgoAdminNoCloudAudit.CreateBuild, FAILED_PRECONDITION'/>
APPLICATION_ERROR;google.devtools.cloudbuild.v1/ArgoAdminNoCloudAudit.CreateBuild;invalid bucket
"staging.<myproject-id-redacted>.appspot.com"; default Cloud Build service account or user-specified service account
does not have access to the bucket;AppErrorCode=9;StartTimeMs=1720838040970;unknown;ResFormat=uncompressed;ServerTimeSec=
1.2677860490000001;LogBytes=256;Non-FailFast;EndUserCredsRequested;EffSecLevel=none;ReqFormat=uncompressed;ReqID=
929904cf53cd5131;GlobalID=0;Server=[2002:a05:6220:101:b0:4:6180:ab19]:4001.
It looks like this has been an issue reported before, with similar posts like this pointing to outages (doesn't seem like there is one reported at this time). Other posts like this one recommend turning on other apis & services and granting additional permissions to service accounts. I have tried all of these recommendations (i.e. turning on additional apis, granting/confirming permissions to the bucket for the service account in question, etc) but to no avail.
Something definitely feels broken and/or there is some missing documentation. Fwiw it's also frustrating that there is no way to get in touch with anyone at Google without paying $30/month for technical support, so I hope someone from the GCP team sees this - either some part of the service is down or the documentation likely needs to be updated.
Per this doc there are some change to App Engine's default service account (namely, on 5/23/2024, Google changed default global permissions for new organizations, in particular disabling iam.automaticIamGrantsForDefaultServiceAccounts).
As of now, these roles required in the default service account to enable gcloud app deploy:
Artifact Registry Create-on-Push WriterStorage AdminLogs Writer