Search code examples
google-cloud-platformpartitioninggoogle-cloud-pubsubaffinity

Subscriber affinity with Ordering Key, but without *in-order* delivery


As far as I have understood, the orderingKey specifies the partition to which a message is published when publishing a message to a particular topic. I have also come to understand that affinity is provided for a given ordering key as to which subscriber instance should receive messages with a specific ordering key. Now my question:

Does a subscription need to enable ordering, i.e. ensure delivery of messages in-order for there to be affinity for an ordering key. My use case does not really require serial delivery of messages of a particular ordering key, but it does require affinity (that all messages with the same ordering key is delivered to the same subscriber instance). In other words, I need a way to ensure that a given partition is only processed by a single subscriber, but I do not care about their respective order within that partition.


Solution

  • Even with ordered delivery enabled, the affinity in Cloud Pub/Sub is best-effort. It is possible for keys to shift among different subscribers as long as there are no messages for that key currently outstanding. This best-effort affinity only exists for subscriptions with ordering enabled. For unordered subscriptions, the ordering key is essentially ignored and messages are delivered to subscribers arbitrarily. Currently, the best way to achieve affinity is to use attributes and filtering, where subscriptions examine the same attribute and look for different values.