Search code examples
google-cloud-platformgoogle-cloud-pubsub

GCP Pub/Sub, can you replay old messages on a new Subscription if there is already an active Subscription


In GCP Pub/Sub I have a topic and a Subscription1 and have have started publishing messages.

Can I add another subscription, Subscription2 and replay old messages that had been published before the Subscription2 was created? Would it allow it? (Kafka allows it)

At what point would I lose access to messages (within the retention period)?, deleting all the Subscriptions? Would keeping at least one active subscription allow PubSub to add new Subscriptions and replay old messages?

Can I also increase the retention period beyond 7 days on the topic?

Edit: The messages will be persisted to DB, but I am more interested in a pub/sub architecture


Solution

  • It would not be good to rely on being able to replay messages in Subscription2 based on the fact that Subscription1 exists. There is no guarantee provided around messages published prior to the existence of Subscription2. The only exception is that if you were to capture a snapshot on Subscription1 and then seek to that snapshot on Subscription2, then you would see the messages published prior to Subscription2's existence. However, you would only see the messages no older than seven days. Therefore, if the snapshot were more than seven days old, you would see no older messages.

    There is no way to extend the retention period beyond 7 days. In general, Kolban's assessment matches the goals of Cloud Pub/Sub: it is a reliable message transport service. The goal of message retention is to ensure that if subscribers are down for a period of time, they will get messages that were sent in that time once they come back up. The same for snapshot and seek: these are particularly useful for the case of restoring from a bad deployment of a subscriber that might have acked messages it should not have. Using seek allows one to replay those messages so they can be processed properly once the subscriber is fixed.