Can I send an alert when a message is published to a pubsub topic?

Question

We are using pubsub & a cloud function to process a stream of incoming data. I am setting up a dead letter topic to handle cases where a message cannot be processed, as described at Cloud Pub/Sub > Guides > Handling message failures.

I've configured a subscription on the dead-letter topic to retain messages for 7 days, we're doing this using terraform:

resource "google_pubsub_subscription" "dead_letter_monitoring" {
  project = var.project_id
  name    = "var.dead_letter_sub_name
  topic   = google_pubsub_topic.dead_letter.name
  expiration_policy { ttl = "" }

  message_retention_duration = "604800s" # 7 days
  retain_acked_messages      = true

  ack_deadline_seconds = 600
}

We've tested our cloud function robustly and consequently our expectation is that messages will appear on this dead-letter topic very very rarely, perhaps never. Nevertheless we're putting it in place just to make sure that we catch any anomalies.

Given the rarity of which we expect messages to appear on the dead-letter-topic we need to set up an alert to send an email when such a message appears. Is it possible to do this? I've taken a look through the alerts one can create at https://console.cloud.google.com/monitoring/alerting/policies/create however I didn't see anything that could accomplish this.

I know that I could write a cloud function to consume a message from the subscription and act upon it accordingly however I'd rather not have to do that, a monitoring alert feels like a much more elegant way of achieving this.

is this possible?

Answer 1

Yes, you can use Cloud Monitoring for that. Create a new policy and perform that configuration

Select PubSub Topic and Published message. Observe the value every minute and count them (aligner in the advanced option). Now, in the config, when it's above 0 from the most recent value, the alert is raised.

To filter on only your topic you can add a filter by topic_id on your topic name.

Then, configure your alert to send an email. It should work!