Terraform : 403 Access denied when using service account impersonalisation and remote state retrieval between different backends

Question

Apologies for the complicated question, I'm still confused about many of its aspects, and it is not even clear to me that I am asking the right interrogations...

I have a GCP terraform project with the following structure :

terraform-project
|--base
  |--main.tf
  |--input.tf
  |--input.tfvars
  |--outputs.tf
|--dev
  |--main.tf
  |--input.tf
  |--input.tfvars
  |--outputs.tf

As you can see, within this infra project I have two environments, base and dev. Within the base environment, I define a remote state bucket in order to handle the lock mechanism etc...

I use impersonation of a service account through a real user (let's call him user_1) in order to create my base infrastructure.

terraform {
  backend "gcs" {
    bucket                      = "base-state-bucket"
    prefix                      = "terraform.tfstate"
    impersonate_service_account = "terraform-super-admin@<redacted_project_id>.iam.gserviceaccount.com"
  }
}

This successfully puts the state in the said bucket in the file : gs://base-state-bucket/terraform.tfstate/default.tfstate

Now within the dev infrastructure, I want to get the base remote state to retrieve certain variables so that I may refer the right folders, projects, etc...

I put the following code in the dev/main.tf:

data "terraform_remote_state" "base" {
  backend = "gcs"
  config = {
    bucket = "base-state-bucket"
    prefix  = "terraform.tfstate"
  }
}

I am still identified as user_1 within gcloud auth login application-default, when I perform terraform plan I get the following error:

data.terraform_remote_state.base: Reading...
╷
│ Error: Error loading state error
│
│   with data.terraform_remote_state.base,
│   on backend.tf line 11, in data "terraform_remote_state" "base":
│   11:   backend = "gcs"
│
│ error loading the remote state: Failed to open state file at gs://base-state-bucket/terraform.tfstetate/default.tfstate:
│ googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access
│ denied.</Message><Details>user_1 does not have storage.objects.get access to the Google Cloud Storage object.</Details></Error>

What's going on here?? The said user_1 is what I used to impersonate my terraform-super-admin, how come he cannot retrieve these files anymore? Is it not impersonating the terraform-super-admin in this block? How do I declare provider=as_terraform_super_admin when this impersonation provider is defined in base backend??

1. I tried to use an actual super administrator user (let's call him super_user_alpha) by logging through gcloud auth login application-default and rerun terraform plan, but what's very very weird is that I still get the 403 Access denied for user_1.
2. When I completely revoke access of user_1 through gcloud auth revoke user_1, I just get a 403 without a mention of which user is denied access, whereas super_user_alpha.. well he is a super user with all the rights on the organisation.

Answer 1

This interrogation was the right one :

How do I declare provider=as_terraform_super_admin when this impersonation provider is defined in base backend??

I kicked myself because it turns out the solution was in the question I asked and within the code itself. It is the use of the config attribute impersonate_service_account

data "terraform_remote_state" "base" {
  backend = "gcs"
  config = {
    bucket = "base-state-bucket"
    prefix  = "terraform.tfstate"
    impersonate_service_account = "terraform-super-admin@<redacted_project_id>.iam.gserviceaccount.com"

  }
}