Prometheus Community Pod in Pending state after installing with helm

Question

I'm following this document (https://aws.amazon.com/blogs/opensource/setting-up-cross-account-ingestion-into-amazon-managed-service-for-prometheus/) to deliver metrics vis AMP to a centralized account with AMG (Amazon Managed Grafana)

My issue is that when i install the prometheus-community/prometheus from helm repo, the alertmanager pod and prometheus-server pod are in pending state. It looks like they require a persistent volume & although i've created it, still same issue.

pv.yaml

    apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: gp2
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  selector:
    matchLabels:
      type: local

values.yaml

serviceAccounts:
    server:
        name: "amp-iamproxy-ingest-service-account"
        annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-ServiceAccount-Role
server:
    remoteWrite:
      - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/${WORKSPACE-ID}/api/v1/remote_write
        queue_config:
            max_samples_per_send: 1000
            max_shards: 200
            capacity: 2500
        sigv4:
            region: us-east-1
            role_arn: arn:aws:iam::${AWS-ACCOUNT-ID}:role/EKS-AMP-Central-Role
    persistentVolume:
      enabled: true
      labels:
        type: local
    emptyDir:
     sizeLimit: ""

Commands:

 kubectl apply -f pv.yaml
    kyubectl apply -f pvc.yaml
    helm install prometheus-for-amp prometheus-

community/prometheus -n prometheus -f ./amp_ingest_override_values.yaml

outputs:

➜ kubectl get pvc -n prometheus                                                                                                                                                                               <aws:sandboxdemo3>
NAME                                        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
prometheus-for-amp-server                   Pending                                      gp2            19s
storage-prometheus-for-amp-alertmanager-0   Pending                                      gp2            18s
task-pv-claim                               Pending                                      gp2            28m

2. kubectl describe pvc prometheus-for-amp-server -n prometheus

     Name:          prometheus-for-amp-server
     Namespace:     prometheus
     StorageClass:  gp2
     Status:        Pending
     Volume:
     Labels:        app=prometheus
                    app.kubernetes.io/managed-by=Helm
                    chart=prometheus-19.7.2
                    component=server
                    heritage=Helm
                    release=prometheus-for-amp
     Annotations:   meta.helm.sh/release-name: prometheus-for-amp
                    meta.helm.sh/release-namespace: prometheus
                    volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
                    volume.kubernetes.io/selected-node: ip-192-168-31-67.ec2.internal
                    volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
     Finalizers:    [kubernetes.io/pvc-protection]
     Capacity:
     Access Modes:
     VolumeMode:    Filesystem
     Used By:       prometheus-for-amp-server-6558f799dd-4bshs
     Events:
       Type    Reason                Age                From                         Message
       ----    ------                ----               ----                         -------
       Normal  WaitForFirstConsumer  30s                persistentvolume-controller  waiting for first consumer to be created before binding
       Normal  ExternalProvisioning  13s (x3 over 29s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
   

   

Answer 1

I was able to fix this by adding "ebs-csi-driver" add on. Looks like cluster didn't have ability to create persistent volume and persistent volume claim by default