Will PubSub forward message to dead letter topic after delivery_attempt exceeds max_delivery_attempts

Question

My subscriber looks like this:

from google.cloud import pubsub_v1
from google.cloud.pubsub_v1.types import DeadLetterPolicy

dead_letter_policy = DeadLetterPolicy(
    dead_letter_topic='dead_letter_topic',
    max_delivery_attempts=5,
)

topic_path = subscriber.topic_path(PROJECT, TOPIC)
subscriber.create_subscription(sub_path, topic_path, dead_letter_policy=dead_letter_policy)

subscriber = pubsub_v1.SubscriberClient()
subscription_path  = subscriber.subscription_path(PROJECT, SUBSCRIPTION)

def callback(message):
    print("Received message: {}".format(message))
    print('Attempted:', message.delivery_attempt, 'times')
    data = message.data.decode('utf-8')
    data_d = json.loads(data)
    if data_d["name"] == "some_file.json":
        message.nack()
    else:
        message.ack()

The received message looks like this:

Received message: Message {
  data: b'{\n  "kind": "storage#object",\n  "id": "...'
  attributes: {
    "bucketId": "sample_bucket",
    ...
  }
}
Attempted: 12 times

Clearly it attempted more than 5 times, why I can still pull this message from PubSub topic? Here is the subscription info:

ackDeadlineSeconds: 10
deadLetterPolicy:
  deadLetterTopic: projects/someproject/topics/dead_letter
  maxDeliveryAttempts: 5
expirationPolicy:
  ttl: 2678400s
messageRetentionDuration: 604800s
name: projects/someproject/subscriptions/new_sub
pushConfig: {}
topic: projects/someproject/topics/pubsub_sample

Answer 1

Typically, this happens if you haven't given Pub/Sub permission to publish to your dead letter topic or subscribe to your subscription. You need to ensure you have run the following:

PUBSUB_SERVICE_ACCOUNT="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"

gcloud pubsub topics add-iam-policy-binding <dead letter topic> \
 --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\
 --role='roles/pubsub.publisher'

gcloud pubsub subscriptions add-iam-policy-binding <subscription with dead letter queue> \
 --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\
 --role='roles/pubsub.subscriber'

If writing to the dead letter queue topic fails, then Cloud Pub/Sub will continue to deliver the message to your subscriber.