How do I set a separate message for warning vs alert in terraform for Datadog?

Question

I'm setting up a monitor that looks like this:

resource "datadog_monitor" "queue_size_critical" {
  message = "High Priority"
  name    = "too many messages in queue"
  query   = "max(last_10m):max:aws.sqs.approximate_number_of_messages_visible{aws_account:<account>,queuename:<queuename>} > 10"
  type    = "metric alert"
  tags    = my_tags
  thresholds = {
    ok = 0
    warning = 1
    critical = 10
  }
  renotify_interval = 1440
}

I've also got a widget that looks like this:

widget {
  alert_value_definition {
    alert_id = datadog_monitor.queue_size_critical.id
    title    = datadog_monitor.queue_size_critical.name
  }
}

I'd like to define two different messages, one of which will be sent when the "warning" threshold is crossed, and the other which will be sent when the "critical" threshold is crossed.

How can I do this?

Is this correct?

resource "datadog_monitor" "queue_size_critical" {
      message = "{{#is_alert}}High Priority{{/is_alert}}
                 {{#is_warning}}Low priority{{/is_warning}}
                 This gets sent every time, in every message."
      name    = "too many messages in queue"
      query   = bla bla bla
      ...etc...
}

Answer 1

That's all handled in the message attribute with conditional logic variables.

If, for example, you define your message value to be this...

{{#is_alert}}
High Priority @pagerduty
{{/is_alert}}

{{#is_warning}}
Medium Priority @slack-mychannel
{{/is_warning}}

You should reference [this dashboard](mydashboardlink) to see how bad this is and follow [these steps](myrunbook) to resolve the situation.

... then ...

1. when your monitor crosses the alert threshold, it will include "High Priority" in the message and it will send a notification to your pagerduty integration,
2. when it crosses the warn threshold it will include "Medium Priority" and send a notification to your slack channel "mychannel", and
3. the "you should reference" part with interesting links would always show up.