Search code examples
google-workflows

Google Cloud Workflows - Concurrent executions limit


For "Concurrent executions", in the list of the "Request limits" of Workflows' quotas docs, there is a limit of "100 executions".

"Concurrent executions - The maximum number of active (started and not yet completed) workflow executions per project: 100 executions"

https://cloud.google.com/workflows/quotas

What does this limit really means, for example if a workflow execution is waiting for a HTTP callback, then it is "started and not yet completed". In this case this limit of 100 executions being in this waiting state seems to me very few.


Solution

  • I am familiar with product internals. This response is quite late, but I can confirm the above understanding is correct. Currently, any execution that has started, but not finished, counts towards Concurrent executions quota. This quota is 7,500 today, and will continue to expand over time. Quota increase requests above this are assessed on a case-by-case basis.

    This quota includes Workflow executions that are:

    • Sleeping
    • Waiting for an HTTP callback
    • all other step types

    For further clarity, Parallel Steps (which spawn multiple branches within an execution) have a separate concurrency limit (See Concurrent branches and iterations here) and do not count towards this quota.

    Requests which exceed Concurrent executions quota will return a 429 error. Clients can retry (recommend using a backoff strategy), or use services like Cloud Tasks, which provides queueing of requests and automatic retries for this type of errors. For variant or unpredictable workloads, it may be worth setting up a Cloud Task queue (highly available, large capacity), offloading the retry management work from your client.

    We will soon preview a feature that queues requests for you, automatically when Concurrent executions quota is exceeded (opt-out available). The advantage of this feature is there is no polling/retry effort or delay from clients or Cloud Tasks. The Workflows service will automatically and immediately start executing the next queued item when a current execution finishes, in (best-effort) FIFO order, maximizing throughput.