I’m trying to install a fresh Prometheus Chart into my AWS EKS cluster using the community helm-chart repo.
However, alertmanager and prometheus-server pods fail to start. They both show an event warning:
Warning FailedScheduling 10m (x25 over 4h13m) default-scheduler running PreBind plugin “VolumeBinding”: binding volumes: timed out waiting for the condition
Our EKS cluster is using EKS managed EC2 nodes, if that’s relevant. Anyone run into this before trying to set up Prometheus on AWS? Any suggestions would be much appreciated!
Steps:
Installed the helm chart via
helm install <chartName> prometheus-community/prometheus -n <namespaceName>
I verified the chart was deployed via
kubectl get pods -n <namespaceName>
Two pods do not start - prometheus-server and alert-manager. The logs from each pod say the same thing:
Warning FailedScheduling 5m22s (x1019 over 7d4h) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
From the error, and default Prometheus chart values for prometheus-server (here) & alertmanager (here), it's waiting for the PVC to be created. Is the EBS CSI driver add-on installed on the cluster? As of EKS 1.23 (changelog here), the in-tree storage plugin has been replaced, which means a CSI driver like the one provided in EBS add-on will be needed in order to provision EBS volumes.