I'm currently working with a system that receives millions of events and I need an efficient way to handle these. My goal is to trigger a cloud function only for certain events, not every single one, to ensure efficient processing and resource utilization.
To achieve this, I was considering using a Pub/Sub model. Specifically, I'm thinking about setting up a Pub/Sub subscription that filters events based on specific criteria relevant to my use case. Then, only if an event matches these criteria, it should trigger a Cloud Function. This way, I can avoid unnecessary invocations of the function and focus only on the relevant events.
My questions are:
Is it possible to trigger a Cloud Function based on a filtered Pub/Sub subscription? If yes, how can I set up this filtered subscription to ensure only specific events trigger the Cloud Function? Are there any best practices or limitations I should be aware of when implementing this approach? I'm aiming for a solution that can handle high throughput efficiently while minimizing unnecessary executions of the Cloud Function. Any insights or guidance on this would be greatly appreciated.
Thanks in advance for your help!
Indeed, when you create a Cloud Function on a topic, a subscription is created but without any filter. And you can't update the subscription filter after the creation. So, in that way you are blocked.
The alternative is not to use the built in mechanism, but to do it manually:
It requires a like rework on your side, but it's more generic by this way.