Search code examples
publish-subscribegoogle-cloud-pubsub

Does recieving message id (future) indicate message is sent in PubSub?


I am sending messages to google pubsub using a callback function which reads back the message id from the future. Using the following code:

"""Publishes multiple messages to a Pub/Sub topic with an error handler."""
import time

from google.cloud import pubsub_v1

# ...

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_name)

def get_callback(f, data):
    def callback(f):
        try:
            print(f.result())
        except:  # noqa
            print('Please handle {} for {}.'.format(f.exception(), data))

    return callback

for i in range(10):
    data = str('message')
    # When you publish a message, the client returns a future.
    future = publisher.publish(
        topic_path, data=data.encode('utf-8')  # data must be a bytestring.
    )
    # Publish failures shall be handled in the callback function.
    future.add_done_callback(get_callback(future, data))

print('Published message with error handler.')

I am receiving back the message id successfully with now errors/exceptions, however I find that some messages are not read into pubsub (when viewing them from the GCP console).

The message id is printed in the line print(f.result()) within the callback function.

My question is: Is it safe to assume that messages are sent to Pubsub successfully following the receipt of a messageid?

If so, what could be the cause for 'dropped' messages?


Solution

  • If publish has returned successfully with a message ID, then yes, Cloud Pub/Sub guarantees to deliver the message to subscribers. If you do not see the message, there are several things that could cause this:

    1. The subscription did not exist at the time the message was published. Cloud Pub/Sub only delivers messages to subscribers if the subscription was created before the message was published.
    2. Another subscriber for the subscription was already running and received the message. If you are using the GCP console to fetch messages while a subscriber is up and running, it is possible that subscriber got the messages.
    3. If you got the message on the console once and then reloaded to get the messages again, you may not see the message again until the ack deadline has passed, which defaults to 10 seconds.
    4. A single pull request (which is what the GCP console "View Messages" feature does), may not be sufficient to retrieve the message. If you click on it multiple times or start up a basic subscriber, you will likely see the message.