Locked out of my terraform state file with 403; How to identify the account performing terraform init?

Question

TL;DR : How do I specify a user account/email for terraform operations such as terraform init ? How do I know which email is being used by default too?

Details :

I have a terraform remote state in a gcp bucket, unfortunately, I got locked out somehow; from the terraform operations, not the organization.

I tested my accesses via gcloud and gsutil using service account impersonation and they seem to be able to read/write to the state bucket via.

gsutil -i "terraform-admin@<project-id>.iam.gserviceaccount.com" cp test-file.txt gs://<state-bucket-id>/terraform.tfstate/test-file.txt

I also verified, the iam.serviceAccountTokenCreator role is well attributed to the user account that is supposed to be impersonating the higher-admin service account.

When I try to do a terraform init, I get :

│ Error: Failed to get existing workspaces: querying Cloud Storage failed: Get "https://storage.googleapis.com/storage/v1/b/<project-bucket-state-for-workspace>/o?alt=json&delimiter=%2F&pageToken=&prefix=terraform.tfstate%2F&prettyPrint=false&projection=full&versions=false": impersonate: status code 403: {
│   "error": {
│     "code": 403,
│     "message": "The caller does not have permission",
│     "status": "PERMISSION_DENIED"
│   }
│ }

I also tried authenticating via gcloud auth login to a user admin account, but I still get the same error. This must mean terraform is not using the right account/email to perform the init..

EDIT : Enabling logs on client side via export TF_LOGS=TRACE

2022-11-21T01:40:23.350+0100 [INFO]  Terraform version: 1.3.4
2022-11-21T01:40:23.351+0100 [DEBUG] using github.com/hashicorp/go-tfe v1.9.0
2022-11-21T01:40:23.351+0100 [DEBUG] using github.com/hashicorp/hcl/v2 v2.14.1
2022-11-21T01:40:23.351+0100 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2022-11-21T01:40:23.351+0100 [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2022-11-21T01:40:23.351+0100 [DEBUG] using github.com/zclconf/go-cty v1.12.0
2022-11-21T01:40:23.351+0100 [INFO]  Go runtime version: go1.19.3
2022-11-21T01:40:23.351+0100 [INFO]  CLI args: []string{"terraform", "init"}
2022-11-21T01:40:23.351+0100 [TRACE] Stdout is a terminal of width 156
2022-11-21T01:40:23.351+0100 [TRACE] Stderr is a terminal of width 156
2022-11-21T01:40:23.351+0100 [TRACE] Stdin is a terminal
2022-11-21T01:40:23.351+0100 [DEBUG] Attempting to open CLI config file: /home/<user>/.terraformrc
2022-11-21T01:40:23.351+0100 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory /home/<user>/.terraform.d/plugins
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory /home/<user>/.local/share/terraform/plugins
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2022-11-21T01:40:23.351+0100 [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2022-11-21T01:40:23.351+0100 [INFO]  CLI command args: []string{"init"}
Initializing modules...
2022-11-21T01:40:23.358+0100 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2022-11-21T01:40:23.363+0100 [DEBUG] Module installer: begin dev-omni-orchestrator-instance
2022-11-21T01:40:23.367+0100 [TRACE] ModuleInstaller: Module installer: dev-omni-orchestrator-instance <nil> already installed in ../modules/omni-orchestrator
2022-11-21T01:40:23.367+0100 [DEBUG] Module installer: begin gcs-infra-genesis-state-buckets
2022-11-21T01:40:23.370+0100 [TRACE] ModuleInstaller: Module installer: gcs-<terraform-project>-state-buckets 3.4.0 already installed in .terraform/modules/gcs-<terraform-project>-state-buckets
2022-11-21T01:40:23.370+0100 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

Initializing the backend...
2022-11-21T01:40:23.382+0100 [TRACE] Meta.Backend: built configuration for "gcs" backend with hash value <V1>
2022-11-21T01:40:23.382+0100 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2022-11-21T01:40:23.382+0100 [DEBUG] New state was assigned lineage "<V2>"
2022-11-21T01:40:23.382+0100 [TRACE] Meta.Backend: moving from default local state only to "gcs" backend
2022-11-21T01:40:23.382+0100 [DEBUG] checking for provisioner in "."
2022-11-21T01:40:23.384+0100 [DEBUG] checking for provisioner in "/usr/bin"
2022-11-21T01:40:23.384+0100 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2022-11-21T01:40:23.384+0100 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2022-11-21T01:40:23.384+0100 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2022-11-21T01:40:23.384+0100 [TRACE] statemgr.Filesystem: read nil snapshot
2022-11-21T01:40:23.384+0100 [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2022-11-21T01:40:23.385+0100 [DEBUG] New state was assigned lineage "<V3>"
╷
│ Error: Failed to get existing workspaces: querying Cloud Storage failed: Get "https://storage.googleapis.com/storage/v1/b/<terraform-project>-terraform-state/o?alt=json&delimiter=%2F&pageToken=&prefix=terraform.tfstate%2F&prettyPrint=false&projection=full&versions=false": impersonate: status code 403: {
│   "error": {
│     "code": 403,
│     "message": "The caller does not have permission",
│     "status": "PERMISSION_DENIED"
│   }
│ }
│

Answer 1

The answer is this :

gcloud auth application-default login

Make sure all of your credentials are redundantly set as per gcloud's arbitrary, badly documented rules.

Here are a few I've used until now to answer for different uses:

gcloud auth login
gcloud auth application-default login
gcloud auth login --no-browser
gcloud config configurations create <...>

I was already checking my authentication status with gcloud auth list that pointed to the right account, but it's not enough. I rarely express frustration or emotion in stackoverflow, but this is quite bad.

This post helped.