Search code examples
elastalert

Turned off Elastalert for a couple days, now its querying all data since I turned it off

I'm testing out elastalert, and there was a strange issue on Wednesday, before the holiday weekend, so I just removed all the alert configurations and rebooted elastalert so it had no alerts over the weekend. Now on Monday I turned it back on and its querying all the logs since Wednesday and its taking quite a long time to catch up. I only want to query recent data. Is this a setting? How do I disable it if I want?

Here is an example alert config:

name: alert-name

type: frequency
num_events: 500
timeframe:
  minutes: 60
realert:
  minutes: 60

index: index.name-*
filter:
- query:
    query_string:
      query: "message:\"Message\" AND context.debug.type.something"

alert_subject: "This alert happened"
alert:
- pagerduty:
    pagerduty_service_key: "nice_try_fbi"
    pagerduty_client_name: "company"

Here is what I'm seeing in the logs (today is 11/26)

INFO:elastalert:Queried rule alert-name from 2018-11-22 12:49 UTC to 2018-11-22 13:04 UTC: 83 / 83 hits
INFO:elastalert:Queried rule alert-name from 2018-11-22 13:04 UTC to 2018-11-22 13:19 UTC: 83 / 83 hits
INFO:elastalert:Queried rule alert-name from 2018-11-22 13:19 UTC to 2018-11-22 13:34 UTC: 89 / 89 hits
INFO:elastalert:Queried rule alert-name from 2018-11-22 13:34 UTC to 2018-11-22 13:49 UTC: 91 / 91 hits
INFO:elastalert:Queried rule alert-name from 2018-11-22 13:49 UTC to 2018-11-22 14:04 UTC: 87 / 87 hits

See how its querying stuff from 4 days ago in 15 minute increments? Timeframe is set to 60 mins. I only want to query the most recent 60 mins. Am I missing something here?

Solution

  • I figured out the answer - looks like remembering the "state" of when the last query was run is a feature of elastalert, as specified here under Reliability: https://github.com/Yelp/elastalert/blob/master/docs/source/elastalert.rst#Reliability

    I also found that it stores the "state" of alerts in indices of the elasticsearch cluster its running on: https://elastalert.readthedocs.io/en/latest/elastalert_status.html

    So I just deleted all of the elastalert* indices in elasticsearch and it seems elastalert treated the alerts as "new" alerts and didn't try to process data since the last successful run. Perhaps there is a better way (like via an alert setting?), but this worked for me.