AWS EKS Auth issues when using different users for Terraform Plan and Apply

Question

I thought it might be cleaner and more secure to use a read-only AWS role to run terraform plan in my CI/CD pipeline. Then a separate more privileged read-write AWS role to terraform apply the plan created earlier. These are separate jobs in the CI/CD pipeline, so there is some separation between plan/read and apply/write operations.

However, I'm facing authentication issues when configuring EKS. As part of the plan phase the Auth data for EKS is created using the read-only role. The read-only role doesn't have any EKS access. So when I re-use the plan in the apply phase it fails to deploy helm charts.

Is it normal to run terraform plan as a read-only role in a CI/CD pipeline?

Should I throw caution to the wind and just use the privileged read-write role for both plan and apply or is there another way to ensure the apply phase uses the correct read-write credentials?

## Get Reference to Kube Cluster
data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

## Get Auth Data
data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

## Use Auth Data
provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

We are using EKS API Cluster authentication mode and have an admin IAM access entry for the read-write role.

This is the error we get from terraform.

Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

If I provide the read-only role admin IAM access entry to EKS, then the error resolves.

Answer 1

After some consideration I decided it was cleaner and simpler to use the same role for both plan and apply stages the my main CI deployment pipeline. Therefore, avoiding any issues with EKS credentials.

I realised that using the read-only role wasn't that much more secure as it would have full read access to the terraform state, including any secrets it contained.

However, using a read-only role for speculative plans on feature branches seems like a good idea. As these should never be executed directly.