Correctly calculate number of sent messages to Dead Letter SQS queue
I want grafana to report how many messages are in a dead letter queue at any given time. The messages that end up within this queue are evaluated and parsed due to the nature of them ending up in this queue.
I've configured grafana to read NumberOfMessagesSent from a dead-letter queue, however, the value is always 0, because; I think messages that end up here are sent from another queue (via SQS).
Even though I can see messages have been sent to the dead-letter queue (not programatially), but by another queue that's configured after X number of receives.
Is there a solution to this?
Quick answer:
Only metric which you can really monitor on SQS DLQ is ApproximateNumberOfMessagesVisible.
TLDR:
When new message arrives to SQS DLQ neither NumberOfMessagesSent or NumberOfMessagesReceived is increased.
Citation from AWS Documentation:
If you send a message to a dead-letter queue manually, it is captured by the NumberOfMessagesSent metric. However, if a message is sent to a dead-letter queue as a result of a failed processing attempt, it isn't captured by this metric. Thus, it is possible for the values of NumberOfMessagesSent and NumberOfMessagesReceived to be different.
NumberOfMessagesSent means:
How many of messages you have sent to the queue (except messages which arrived to DLQ as result of failure)
NumberOfMessagesReceived means:
How many of messages you have received from the queue
ApproximateNumberOfMessagesVisible means:
Total number or messages, which are visible in the queue. (Remember when you receive message from the queue you have to set visibility timeout == how long message you have received is not visible for others. Also remember you should delete message after successful processing, otherwise it will be after visibility timeout available to others.)