Search code examples
elasticsearchelastic-stackelastalert

Unable to access data inside alert section of elastalert


I have been trying to set up elastalert monitoring on my ELK stack. For the beginning I want to set up a simple rule which will generate a notification if any disk on the file system has reached 80% usage. The rule seems to be working correctly but in the alert section I am not able to pass the data to python script. The uncommented command in the alert section gives following error

ERROR:root:Error while running alert command: Error formatting command: 'system.filesystem.mount_point' error.

Here is my rule file. Please excuse the formatting of the yaml.

name: Metricbeat high FS percentage
type: metric_aggregation

es_host: localhost
es_port: 9200

index: metricbeat-*

buffer_time:
minutes: 1

metric_agg_key: system.filesystem.used.pct
metric_agg_type: max
query_key: beat.name.keyword
doc_type: metricsets

bucket_interval:
  minutes: 1

realert:
  minutes: 2

sync_bucket_interval: true
#allow_buffer_time_overlap: true
#use_run_every_query_size: true

max_threshold: 0.8

filter:
- query:
    query_string:
      query: "system.filesystem.device_name: dev"
      analyze_wildcard: true
- term:
    metricset.name: filesystem

# (Required)
# The alert is use when a match is found

alert:
  - debug 
  - command
command: ["/home/ubuntu/sendToSlack.py","beat-name","%(beat.name.keyword)s","used_pc","%(system.filesystem.used.pct_max)s","mount_point","%(system.filesystem.mount_point)s"]
 # command: ["/home/ubuntu/sendToSlack.py","--beat-name","{match[beat.name.keyword]}","--mount_point","{match[system.filesystem.mount_point]}"]
 # command: ["/home/ubuntu/sendToSlack.py","--beat-name","{match[beat][name]}","--mount_point","{match[system][filesystem][mount_point]}"]
 #pipe_match_json: true
 #- command:
 #    command: ["/home/ubuntu/sendToSlack.py","%(system.filesystem.used.bytes)s"]

Some observations: On testing the rule file using the command python -m elastalert.test_rule rules/high_fs.yaml I get the output

Successfully loaded Metricbeat high FS percentage

Got 149161 hits from the last 1 day

Available terms in first hit:

tags
beat.hostname
beat.name
beat.version
type
@timestamp
system.filesystem.available
system.filesystem.files
system.filesystem.mount_point
system.filesystem.free_files
system.filesystem.free
system.filesystem.device_name
system.filesystem.used.bytes
system.filesystem.used.pct
system.filesystem.total
host
@version
metricset.rtt
metricset.name
metricset.module

I should be able to access any of the fields mentioned above. When I run this rule using python -m elastalert.elastalert --verbose --rule rules/high_fs.yaml a list is printed on the screen

@timestamp: 2017-10-18T17:15:00Z
beat.name.keyword: my_server_name
num_hits: 98
num_matches: 5
system.filesystem.used.pct_max: 0.823400020599

I am able to access all the key value pairs in this list. Anything thats outside the list fails with the formatting error. Been stuck over this for long. Any help is appreciated.


Solution

  • UPDATE: A reply for the same problem on elastalert's github repo says that certain query types do not contain the full field data.

    While I am not sure if this is the correct way to achieve what I was looking for but I was able to get the the desired output using the rule type any and writing my own filters. Here is how one of my rules file looks currently.

    name: High CPU percentage
    type: any
    
    es_host: localhost
    es_port: 9200
    
    index: consumer-*
    query_key:
      - beat.name
    
    filter:
    - range:
        system.cpu.total_norm_pct:
          from: 0.95
          to: 10.0
    
    realert:
      minutes: 60
    
    alert:
    - command:
        command: ["/home/ubuntu/slackScripts/sendCPUDetails.py","{match[beat][name]}","{match[system][cpu][total_norm_pct]}"]
    new_style_string_format: true
    

    Hope it helps someone.