How to create aws-ebs-csi-driver with eks_blueprints_addons by Terraform?

Question

I created AWS EBS CSI Driver addon with eks_blueprints_addons by Terraform:

    module "eks_cluster" {
      source  = "terraform-aws-modules/eks/aws"
      version = "~> 20.17"
      ...
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent = true
        }
      }
    }

I can confirm it created a ebs driver in the kube-system namespace. When I deploy an application with PVC, I got this error:

Warning ProvisioningFailed 113s (x2 over 4m53s) ebs.csi.aws.com_ebs-csi-controller-c4bc5f559-k6fqp_5a4ed8cc-0085-4875-8070-87fceda36abf (combined from similar events): failed to provision volume with StorageClass "standard": rpc error: code = Internal desc = Could not create volume "pvc-f8f034de-5f7b-4ca6-bb0c-0c3e4be8026d": could not create volume in EC2: operation error EC2: CreateVolume, https response error StatusCode: 403, RequestID: 0d5d6201-4115-4cd9-bc99-201331b97450, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::111111111111:assumed-role/default-eks-node-group-2024071204291580710000000e/i-075c478df65c9bd58 is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:ap-northeast-1:111111111111:volume/* because no identity-based policy allows the ec2:CreateVolume action 

It seems using an assume role created by EKS module. Even I use this way got the same result:

    module "ebs_csi_driver_irsa" {
      source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
      version = "~> 5.20"

      role_name_prefix = "ebs-csi-driver-"

      attach_ebs_csi_policy = true

      oidc_providers = {
        main = {
          provider_arn               = module.eks_cluster.oidc_provider_arn
          namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
        }
      }

      tags = var.tags
    }

    module "eks_blueprints_addons" {
      source  = "aws-ia/eks-blueprints-addons/aws"
      version = "~> 1.1"

      cluster_name      = module.eks_cluster.cluster_name
      cluster_endpoint  = module.eks_cluster.cluster_endpoint
      cluster_version   = module.eks_cluster.cluster_version
      oidc_provider_arn = module.eks_cluster.oidc_provider_arn

      eks_addons = {
        aws-ebs-csi-driver = {
          most_recent              = true
          service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
        }
      }
    }

But if I do with eksctl, it works:

    eksctl create addon --name aws-ebs-csi-driver --cluster eks-test --service-account-role-arn arn:aws:iam::11111111:role/AmazonEKS_EBS_CSI_DriverRole --force

    eksctl create iamserviceaccount \
        --name ebs-csi-controller-sa \
        --namespace kube-system \
        --cluster eks-test \
        --role-name AmazonEKS_EBS_CSI_DriverRole \
        --role-only \
        --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
        --approve

Answer 1

Have you tried attaching AmazonEBSCSIDriverPolicy to your role created by ebs_csi_driver_irsa ?

I am having similar issue, it looks like module "eks_blueprints_addons" and eksctl have different approach in creating service accounts.

Also, you may want to try aws_ebs_csi_pod_identity, but you would need Amazon EKS Pod Identity Agent addon for that

module "aws_ebs_csi_pod_identity" {
  source = "terraform-aws-modules/eks-pod-identity/aws"

  name = "aws-ebs-csi-pod-identity"

  attach_aws_ebs_csi_policy = true

  association_defaults = {
    namespace       = "kube-system"
    service_account = "ebs-csi-controller-sa"
  }
  associations = {
    eks = {
      cluster_name = aws_eks_cluster.this.name
    }
  }

  tags = {
    Environment = "${var.env}"
  }
}