How to monitor disk usage of persistent volumes of all storageclasses using Prometheus?

Question

I am using Kubernetes Persistent Volumes dashboard on Grafana to view the disk usage for my PVCs. This dashboard makes use of kubelet_volume_stats_used_bytes metric to fetch the data for the PVCs and I am able to visualize it as well.

But, it does not display the PVCs which use EFS as storageclass. I found some stackoverflow answers and comments regarding the same, but none of them included a solution.

How to monitor kubernetes persistence volume claim i.e disk usage

How to monitor disk usage of kubernetes persistent volumes?

So my question is how do get usage metrics for EFS PVCs? Or to make it more generic - how to scrape PVC metrics for all storageclasses?

Answer 1

EFS PVCs don't have any size limits. Thats why this traditional method doesn't work while checking their disk usage. Instead, I was able to use a sidecar-logging mechanism to get this data -

Attach the following sidecar to the pod which is using the PVC myPvcName

      containers:
      - name: pvc-logging
        image: busybox:latest
        command: [ "/bin/sh", "-c", "--" ]
        args: [ "while true; do du -sh /myPvc ; sleep 5; done;" ]
        volumeMounts:
        - mountPath: /myPvc
          name: myPvc
          readOnly: true
      volumes:
        - name: myPvc
          persistentVolumeClaim:
            claimName: myPvcName

This sidecar -

• has the MyPvcName pvc mounted at the path /myPvc
• checks the space used by this pvc every 5 seconds (you can change this interval)

It's logs look like -

10.3G   /myPvc
10.3G   /myPvc
10.4G   /myPvc

These can be parsed using whatever logging stack you use - in my case, it was the elastic stack.

But I haven't yet looked into how I can send these logs into prometheus so I can visualise them on Grafana.