We are developing a service that stores internal data in GCP Firestore. For some changes in data, we need to send messages to external services.
To ensure reliability we want to send messages if and only if the transaction that changes our internal data is committed.
To achieve this goal we gonna use the transactional outbox pattern. So we created a Firestore collection to which we append new events/messages as documents. Because Firestore has no concept of transactions WAL (at least as Public API) we cannot use transaction log tailing pattern. So that the only option that we have is polling-publisher pattern. But this leads to a lack of message ordering because we cannot determine which of events in the transactional-outbox collection was created firstly. Please take into attention that just a document timestamp on the application side is not an option, because it does not tell us which transaction was committed first.
Note: We can use cloud functions to get updates about new documents in the transactional-outbox collection, but it does not guarantees ordering as well.
Note 2: We use GCP PubSub as a queue where we propagate events from the transactional-outbox collection.
The question is: are there any features in GCP Firestore that we miss and that can help us to propagate events to PubSub topics in the same order as they were created in the Firestore collection?
According to the Firestore documentation, when your system creates a document, you can attach in it a server timestamp. This timestamp can help you in establishing an order within the documents based on creation:
Important: Unlike "push IDs" in the Firebase Realtime Database, Cloud Firestore auto-generated IDs do not provide any automatic ordering. If you want to be able to order your documents by creation date, you should store a timestamp as a field in the documents
The next step would be to create logic in a Cloud Function that publishes documents to Pub/Sub based on the timestamp. The order of publishing messages to Pub/Sub and the order in which subscribers received them can be preserved by using Pub/Sub ordering. Pub/Sub ordering is based on an ordering key that associates a number of messages with an entity, and Pub/Sub orders the delivery based on the publish order.
Based on your requirement of the transactional outbox following chronological order, this could achieve that purpose.