Search code examples
elasticsearchkubernetesmetricbeat

Kubernetes Percentage Used of a Volume with Metricbeat

I am using metricbeat to gather metrics about my k8s cluster and the pods running within it. I would like to setup alerting for my PVCs and to that end I need to know the percentage of a PVC that is used.

From metricbeat I am getting data regarding volumes and their total capacity, amount used... however there is no field for the percentage used.

I have seen articles where I can use the visual builder to 'compute' the percentage but I am not sure that helps me with alerts. It seems that I need a field with the value of percentage used.

I cannot find a way to configure metricbeat to create a new field for the percentage used. It does seem there is an option using a scripted field but this seems to carry performance implications.

  1. How can I setup alerts for percentage used of k8s volumes?
  2. What is the best way to get an index field for percentage used of a volume?
Solution

  • There were two options that I found:

    1. Scripted fields in Kibana
    2. Update Logstash Indexer to do the computation and add a field to the index

    I ultimately went with #2 and added a ruby filter to my indexer config to compute the field and add it.

    filter {
  ruby {
     code => ' total = event.get("[kubernetes][volume][fs][capacity][bytes]");
            if (total != nil)
               used = event.get("[kubernetes][volume][fs][used][bytes]");
               percentUsed = (used.to_f / total)
               event.set("kubernetes.volume.fs.percentage.used", percentUsed)
            end'
  }

    }

    I selected #2 because I did not want to burden Kibana and I had more capacity on the indexers.