Search code examples
amazon-web-servicesamazon-cloudwatchamazon-sqsamazon-sns

How to ensure a SNS topic has a subscription?


I have a stack defined in cdk that consists of an SNS topic in one AWS account and a SQS queue subscribed to the topic in another.

I am worried that there could be a case where the subscription fails or is temporary unavailable (deployments/failed deployments), in which messages would be lost. If the subscription fails the message would be published to the topic but deleted silently since nothing would be subscribed.

This seems like a gap considering SNS + SQS in another AWS account would be a common design pattern.

Questions:

  1. How can we monitor a subscription on a topic? I suppose you could write a scheduled lambda that polls the subscriptions to a queue and emits the metric to cloudwatch and alarm on that but this seems impractical. Is this something we should be worried about? (deletion protection is enabled on the resources). I was thinking of alarming on (NumberOfMessagesPublished - NumberOfNotificationsDelivered) but if this alarm was triggered, there would be no way of recovering the message unless the message was recorded elsewhere, which would require we log every message since no explicit failures took place.
  2. Is subscription unavailability during deployments something we should worry about?

Solution

  • What I ended up doing was adding an alarm on a Math expression NumberOfMessagesPublished - NumberOfNotificationsDelivered and logging all messages at the source of the event so they are recoverable in the case the subscription is lost.

    This is ok for our usecase since this event does not happen a lot so the logging wouldn't be that bad for performance/cost.