Scaling Google Cloud Pull subscribers

Question

I'm considering Google Cloud Pub/Sub and trying to determine whether I will go for Pull or Push subscriber model.

Notably the pull model is able to handle larger throughput

• Large volume of messages (many more than 1/second)

• Efficiency and throughput of message processing is critical.

However, the push model can sit behind an HTTP load balancer, and is therefore able to auto-scale subscriber nodes during times when the number of queued messages exceeds the capacity of a single subscriber node.

The pull model is also more secure, because it does not require exposing sensitive operations in an HTTP endpoint.

The issue is how can we scale subscriber nodes in the pull model? Is there something in GCP for this kind of situation?

Answer 1

There are several options for auto scaling via a pull subscriber:

• Use GKE and to set up Autoscaling Deployments with Cloud Monitoring Metrics. This allows you to scale the number of instances based on the num_undelivered_messages metric.
• Use a GCE managed instance group and scale based the num_undelivered_messages metric.
• Use Dataflow to process messages from Pub/Sub and set up autoscaling.