Logstash metric filter for log-level

Question

Can someone please help me with my metric filter. i want to set up logstash to check the log-level= Error for every 5s and if log-level = ERROR exceeds more than 1 , should send an email. i am using logstash 2.2.4

input {
      file {
        path => "/var/log/logstash/example"
        start_position => beginning
      }
    }

filter {
 grok{
   match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{LOGLEVEL:log-level}\s*\]" }
 }
 if [log-level] == "ERROR" {
   metrics {
       meter => [ "log-level" ]
       flush_interval => 5
       clear_interval => 5
      }
   }
}
output {
if [log-level] == "ERROR" {
  if [log-level][count] < 1 {
    email {
        port => 25
        address => "mail.abc.com"
        authentication => "login"
        use_tls => true
        from => "[email protected]"
        subject => "logstash alert"
        to => "[email protected]"
        via => "smtp"
        body => "here is the event line %{message}"
        debug => true
      }
    }
  }
}

Answer 1

Editorial:

I am not a fan of the metrics {} filter, because it breaks assumptions. Logstash is multi-threaded, and metrics is one of the filters that only keeps state within its thread. If you use it, you need to be aware that if you're running 4 pipeline workers, you have 4 independent threads keeping their own state. This breaks the assumption that all events coming "into logstash" will be counted "by the metrics filter".

For your use-case, I'd recommend not using Logstash to issue this email, and instead rely on an external polling mechanism that hits your backing stores.

---

Because this is the metrics filter, I highly recommend you set your number of filter-workers to 1. This is the -w command-line option when logstash starts. You'll lose parallelism, but you'll gain the ability for a single filter to see all events. If you don't, you can get cases where all, say, 6 threads each see an ERROR event; and you will get six emails.

Your config could use some updates. It's recommended to add a tag or something to the metrics {} filter.

 metrics {
     meter => [ "log-level" ]
     flush_interval => 5
     clear_interval => 5
     add_tag => "error_metric"
    }
 }

This way, you can better filter your email segment.

output {
  if [tags] include "error_metric" and [log-level][count] > 1 {
    email {
    }
  }
}

This is because the metrics {} filter creates a new event when it flushes, rather than amending an existing one. You need to catch the new event with your filters.