Prometheus - Expression under annotations

Question

I want to send an email alert (I have a custom template) that looks like this:

Description = Disk is almost full: < 20% left

Summary = Volume D: on 192.168.1.10 - Windows Server Disk Space Usage
Size = 200Gi
Used = 180Gi
Available = 20Gi

So far I have:

alert: 'WARNING - Host out of Disk Space'
expr: 100.0 - 100 * ((windows_logical_disk_free_bytes / 1024 / 1024 ) / (windows_logical_disk_size_bytes / 1024 / 1024)) > 80
for: 1m
labels:
  severity: warning
annotations:
  Summary: "Volume {{ $labels.volume }} on {{ $labels.instance }} - Windows Server Disk Space Usage"
  Description: "Disk is almost full: < 20% left)"
  Size: "{{ printf \"windows_logical_disk_size_bytes{volume='%s'}\" .Labels.volume | query | first | value | humanize1024 }}"
  Used: "{{ printf \"windows_logical_disk_free_bytes{volume='%s'}\" .Labels.volume | query | first | value | humanize1024 }}"
  Available: ???

I have tried a few options for "Available" but none of them work. I get 'error expanding template'. I basically want this query => (windows_logical_disk_size_bytes{volume='%s'} - windows_logical_disk_free_bytes{volume='%s'})

What would be the right way to do it? Any help would be appreciated.

Answer 1

First of all: the correct expressions for "Used" and "Available" are the following ones, aren't they?

Used:      windows_logical_disk_size_bytes - windows_logical_disk_free_bytes
Available: windows_logical_disk_free_bytes

Answering your question:

I think you can't use expressions in alert templates then you need to define a "recording rule" like the following:

  - name: recording_rules
    rules:
      - record: windows_filesystem_used
        expr: windows_logical_disk_size_bytes - windows_logical_disk_free_bytes

See more info in the Prometheus documentation here.