I am trying to use Persistent Volumes in AWS EKS.
I followed all the steps mentioned from the below link
https://aws.amazon.com/premiumsupport/knowledge-center/eks-persistent-storage/
When I deployed sample pods to check PVCs are created fine, I am getting below error.
Warning FailedScheduling 2m6s (x265 over 44h) default-scheduler error while running "VolumeBinding" prebind plugin for pod "app": Failed to bind volumes: timed out waiting for the condition
Checked further persistent volume claims
kubectl describe pvc
Warning ProvisioningFailed 62s (x7 over 3m5s) ebs.csi.aws.com_ebs-csi-controller-7cb88dcffc-vkvdk_2ae45c3c-b4f2-4200-a5d4-cfdb35a5f0db failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
From looking at the logs of ebs csi controller
I0721 10:44:10.049138 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"avengers", Name:"ebs-claim", UID:"88a9684c-1be7-412f-8616-b662eff692df", APIVersion:"v1", ResourceVersion:"19320908", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = Internal desc = WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-2.amazonaws.com/id/XXXXXXXXXXXX34FDCBXXXXX
status code: 400, request id: 90b430ba-6999-41d8-ac15-13f5ec73def2
I0721 10:44:45.207103 1 controller.go:1332] provision "default/ebs-claim" class "ebs-sc": started
I0721 10:44:45.207406 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3922ab97-483a-486a-bbbd-57875e506021", APIVersion:"v1", ResourceVersion:"19875393", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/ebs-claim"
I0721 10:44:55.207594 1 controller.go:1106] Temporary error received, adding PVC 3922ab97-483a-486a-bbbd-57875e51 to claims in progress
W0721 10:44:55.207639 1 controller.go:958] Retrying syncing claim "3922ab97-483a-486a-bbbd-57875e506021", failure 9
E0721 10:44:55.207664 1 controller.go:981] error syncing claim "3922ab97-483a-486a-bbbd-57875e506021": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0721 10:44:55.207859 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3922ab97-483a-486a-bbbd-57875e506021", APIVersion:"v1", ResourceVersion:"19875393", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context
What did I miss here ?
Any help will be appreciated.
From the error log, pick the id for search and i.e. XXXXXXXXXXXX34FDCBXXXXX
caused by: InvalidIdentityToken: No OpenIDConnect provider
found in your account for
https://oidc.eks.us-east-2.amazonaws.com/id
/XXXXXXXXXXXX34FDCBXXXXX
status code: 400, request id: 90b430ba-6999-41d8-ac15-13f5ec73def2
IAM OIDC (OpenID Connect) Identity provider didn’t exist for current EKS Cluster.
aws iam list-open-id-connect-providers | grep XXXXXXXXXXXX34FDCBXXXXX
returned nothing.
Enabled OIDC IAM in existing EKS Cluster:
$ eksctl utils associate-iam-oidc-provider --cluster=eks-lab --region us-east-2 --approve
Deleted everything and deployed again and now it works fine.