Terraform on GCP - Impersonation not working (missing permission in generating access token)

Question

I'm trying to create a CI/CD pipeline from the Terraform scripts used in my organization. The pipeline works perfectly when executed locally (using an high privileged account), but they don't when executed remotely.

I think it's a permission issue, but I cannot spot where.

I created a SA for Terraform and I granted it these roles:

resource "google_organization_iam_member" "org_tf_folder_admin" {
  count  = var.folder_id != "" ? 0 : 1
  org_id = var.org_id
  role   = "roles/resourcemanager.folderAdmin"
  member = "serviceAccount:${module.terraform_project.terraform_sa_email}"
}

resource "google_organization_iam_member" "org_tf_project_creator" {
  count  = var.folder_id != "" ? 0 : 1
  org_id = var.org_id
  role   = "roles/resourcemanager.projectCreator"
  member = "serviceAccount:${module.terraform_project.terraform_sa_email}"
} 

I can see the roles are currently applied to the account.

Now, in versions.tf I asked Terraform to impersonate the Service Account

provider "google" {
  impersonate_service_account = ${module.terraform_project.terraform_sa_email}
}

provider "google-beta" {
  impersonate_service_account = ${module.terraform_project.terraform_sa_email}
}

But I get a 403 error when I try, for example, to list the folders inside the organization, or to access a project inside the var.folder_id folder. This is true when executed locally and also inside CloudBuild, so I think it's a permission issue, rather than a SA issue (but of course I cannot be sure). Perhaps some inheritance?

This is the error I get:

│ Error: Error when reading or editing Folder Not Found : folders/XXXXXXXXXXXX: Get "https://cloudresourcemanager.googleapis.com/v3/folders/XXXXXXXXXXXX?alt=json&prettyPrint=false": impersonate: status code 403: {
│   "error": {
│     "code": 403,
│     "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).",
│     "status": "PERMISSION_DENIED",
│     "details": [
│       {
│         "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│         "reason": "IAM_PERMISSION_DENIED",
│         "domain": "iam.googleapis.com",
│         "metadata": {
│           "permission": "iam.serviceAccounts.getAccessToken"
│         }
│       }
│     ]
│   }
│ }
│
│
│   with google_folder.env,
│   on main.tf line 37, in resource "google_folder" "env":
│   37: resource "google_folder" "env" {

Edit

The cloudbuild default service account can actually impersonate:

resource "google_service_account_iam_member" "cloudbuild_terraform_sa_impersonate_permissions" {
  service_account_id = var.terraform_sa_name
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = "serviceAccount:${module.cloudbuild_project.project_number}@cloudbuild.gserviceaccount.com"
}

but as I said, since it doesn't work locally too (when I execute using my user identity), the issue should be elsewhere

Any idea?

Edit 2

I added a block to explicitly create the accessToken, but nothing changed. I'm running out of options, I must admit

provider "google" {
  alias = "tokengen"
}

# get config of the client that runs
data "google_client_config" "default" {
  provider = google.tokengen
}

data "google_service_account_access_token" "sa" {
  provider               = google.tokengen
  target_service_account = local.tf_sa
  lifetime               = "600s"
  scopes = [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/userinfo.email",
  ]
}

/******************************************
  GA Provider configuration
 *****************************************/

provider "google" {
  access_token = data.google_service_account_access_token.sa.access_token
}
provider "google-beta" {
  access_token = data.google_service_account_access_token.sa.access_token
}

Answer 1

A 403 Permission denied error such as Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist) is due to the principalSet on the service account IAM binding not matching the principalSubject making the call. This means the service account that you are impersonating does not have the appropriate permission to access the intended resource. This is a permission issue on the resources, instead of being an impersonation issue.

To create or access a Token you need to grant the permission to impersonation service account iam.serviceAccounts.getAccessTokenand Service Account Token Creator role . Try adding above permissions and have a try .

Refer to this service account impersonation and Blog by Tanuj Bolisetty.