Search code examples
bosun

Bosun: only first notification being sent

I am experiencing strange behaviour with my first-time Bosun config. It seems I'm receiving mail only first time the alert is being called. I am surely misunderstanding something.. Here's my .conf file:

smtpHost = ${env.BOSUN_MAIL_SERV}
smtpUsername =  ${env.BOSUN_MAIL_USER}
smtpPassword = ${env.BOSUN_MAIL_PASS}
emailFrom = ${env.BOSUN_MAIL_FROM}
httpListen = :8070
timeAndDate = 31
stateFile = ../bosun.state
ledisDir = ../ledis_data
checkFrequency = 10s
logstashElasticHosts = ${env.BOSUN_ES_HOSTS}

notification onError {
    email = [email protected]
    print = true
}

template generic {
    subject = Test alert
    body = `<p>Alert: {{.Alert.Name}} triggered </p>`
}

alert company.errors.rate.exceeds0 {
    template = generic
    unknownIsNormal = true
    runEvery = 1
    $notes = "The rate of errors exceeds 1."
    crit = 1
    critNotification = onError
}

As you can see, crit is set always. However, nothing never happens. The Bosun output is like this:

elkarel@thinkPad-E550:~/Bosun$ ./bosun-linux-amd64-0.5.0-rc3 -c company.conf 
2016/05/18 13:14:45 enabling syslog
2016/05/18 13:14:45 info: search.go:194: Loading last datapoints from redis
2016/05/18 13:14:45 info: search.go:201: Done
2016/05/18 13:14:45 info: bolt.go:53: RestoreState
2016/05/18 13:14:45 info: bolt.go:65: RestoreState done in 23.954µs
2016/05/18 13:14:45 info: check.go:513: check alert company.errors.rate.exceeds0 start
2016/05/18 13:14:45 info: check.go:537: check alert company.errors.rate.exceeds0 done (463.184µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:14:45 info: web.go:154: bosun web listening on: :8070
2016/05/18 13:14:45 info: web.go:155: tsdb host: 
2016/05/18 13:14:45 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds0 took 11.525239ms
2016/05/18 13:14:55 info: check.go:513: check alert company.errors.rate.exceeds0 start
2016/05/18 13:14:55 info: check.go:537: check alert company.errors.rate.exceeds0 done (418.716µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:14:55 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds0 took 2.298315ms

2016/05/18 13:15:05 info: notify.go:147: Batching and sending unknown notifications
2016/05/18 13:15:05 info: notify.go:177: Done sending unknown notifications
2016/05/18 13:15:05 info: check.go:513: check alert company.errors.rate.exceeds0 start
2016/05/18 13:15:05 info: check.go:537: check alert company.errors.rate.exceeds0 done (380.23µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:15:05 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds0 took 919.933µs
2016/05/18 13:15:15 info: check.go:513: check alert company.errors.rate.exceeds0 start
2016/05/18 13:15:15 info: check.go:537: check alert company.errors.rate.exceeds0 done (351.148µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:15:15 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds0 took 807.104µs
2016/05/18 13:15:25 info: notify.go:147: Batching and sending unknown notifications
2016/05/18 13:15:25 info: notify.go:177: Done sending unknown notifications
2016/05/18 13:15:25 info: check.go:513: check alert company.errors.rate.exceeds0 start
2016/05/18 13:15:25 info: check.go:537: check alert company.errors.rate.exceeds0 done (439.26µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:15:25 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds0 took 2.429369ms

So he knows the alert resulted in a critical state, but does not send any notification. However, if I change alert name to company.errors.rate.exceeds0_newname, the notification is fired and mail arrives:

elkarel@thinkPad-E550:~/Bosun$ ./bosun-linux-amd64-0.5.0-rc3 -c company.conf 
2016/05/18 13:18:35 enabling syslog
2016/05/18 13:18:35 info: search.go:194: Loading last datapoints from redis
2016/05/18 13:18:35 info: search.go:201: Done
2016/05/18 13:18:35 info: bolt.go:53: RestoreState
2016/05/18 13:18:35 info: bolt.go:65: RestoreState done in 28.431µs
2016/05/18 13:18:35 info: check.go:513: check alert company.errors.rate.exceeds_newname start
2016/05/18 13:18:35 info: check.go:537: check alert company.errors.rate.exceeds_newname done (638.843µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:18:35 info: web.go:154: bosun web listening on: :8070
2016/05/18 13:18:35 info: web.go:155: tsdb host: 
2016/05/18 13:18:35 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds_newname took 2.380886ms
2016/05/18 13:18:35 info: notify.go:57: Test alert
2016/05/18 13:18:36 info: notify.go:115: relayed alert company.errors.rate.exceeds_newname{} to [[email protected]] sucessfully. Subject: 10 bytes. Body: 99 bytes.
2016/05/18 13:18:45 info: check.go:513: check alert company.errors.rate.exceeds_newname start
2016/05/18 13:18:45 info: check.go:537: check alert company.errors.rate.exceeds_newname done (558.396µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:18:45 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds_newname took 746.837µs
2016/05/18 13:18:55 info: notify.go:147: Batching and sending unknown notifications
2016/05/18 13:18:55 info: notify.go:177: Done sending unknown notifications
2016/05/18 13:18:55 info: check.go:513: check alert company.errors.rate.exceeds_newname start
2016/05/18 13:18:55 info: check.go:537: check alert company.errors.rate.exceeds_newname done (394.645µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:18:55 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds_newname took 956.957µs
2016/05/18 13:19:05 info: check.go:513: check alert company.errors.rate.exceeds_newname start
2016/05/18 13:19:05 info: check.go:537: check alert company.errors.rate.exceeds_newname done (404.464µs): 1 crits, 0 warns, 0 unevaluated, 0 unknown
2016/05/18 13:19:05 info: alertRunner.go:55: runHistory on company.errors.rate.exceeds_newname took 832.393µs
2016/05/18 13:19:15 info: notify.go:147: Batching and sending unknown notifications

Notifications are being sent apparently only with "new" alerts. Is this how Bosun works? Do I have to resend the notification until somebody acks it?

The mail server is gmail.

Thanks in advance!

Solution

  • Yes, Bosun only sends notification when an alert transitions from normal to warn or from warn to crit. You can, however chain notifications or even specify

    notification onError {
    email = [email protected]
    print = true
    timeout = 1m
    next = onError
}

    That way Bosun will send notification every minute until the alert gets acknowledged.