How to use Google Monitoring Query Language to fetch metrics from different resource types?

Question

I am trying to create an alert on Google Cloud, to be notified about Pub/Sub messages which cannot be forwarded to a dead-letter topic, using Google Monitoring Query Language.

On Google documentation, it is said:

To verify that Pub/Sub is forwarding undeliverable messages, you can compare the subscription/dead_letter_message_count metric with the topic/send_message_operation_count metric from the topic that Pub/Sub forwards these messages to.

However, when trying to create an alert condition with the Query Editor, I get the following error:

Metric 'pubsub.googleapis.com/topic/send_message_operation_count' is not compatible with resource 'pubsub_subscription'. Consider using the following resource(s): 'pubsub_topic'.

I am wondering then if it is possible to fetch topic and subscription metrics with the Google Monitoring Query Language, and if yes, how?

Answer 1

You can fetch both topic and subscription metrics in the Query Editor, although you need to specify the correct resource type for each metric. You can fetch pubsub.googleapis.com/topic/send_message_operation_count with:

fetch pubsub_topic
  | metric 'pubsub.googleapis.com/topic/send_message_operation_count'

And pubsub.googleapis.com/subscription/dead_letter_message_count with

fetch pubsub_subscription
  | metric 'pubsub.googleapis.com/subscription/dead_letter_message_count'

In order to use both metrics together in a single alert, you will need to use a join.

Here's an example of how you can configure an alert verifying that the rate of publish never falls below the rate of dead-lettering by more than 1 QPS/min:

{
  fetch pubsub_subscription
  | metric 'pubsub.googleapis.com/subscription/dead_letter_message_count'
  | filter resource.subscription_id = "my-sub"
  | group_by [], sum(val())
  | align rate(1m)
  | every 1m
;
  fetch pubsub_topic
  | metric 'pubsub.googleapis.com/topic/send_message_operation_count'
  | filter resource.topic_id = "my-dead-letter-topic"
  | group_by [], sum(val())
  | align rate(1m)
  | every 1m
}
| join
| sub
| condition val() > 1 # 1/min