Using cloud functions to schedule cloud tasks, upon scheduled time the cloud task triggers an HTTP end point. As of now created a single queue with the following configuration. Max dispatches per second:500 Max concurrent dispatches :1000 Max attempts: 5
The cloud function is pub sub triggered. In a second pub sub may be receiving 10000 messages and in turn the cloud function scales and will be creating 10000 tasks.
Question: If the scaled cloud functions has to create more tasks and assign it to different queues , how best the cloud function has to decide and create queues and assign tasks to different queues considering cold and warm queues capabilities to avoid latency.
I read through this official doc, but it is not so clear for dummies https://cloud.google.com/tasks/docs/manage-cloud-task-scaling#queue
Back to your original question, if your process is time sensitive and you need to trigger more than 500 requests at the same time, you need to create additional queues (as mentioned in the documentation)
To dispatch the AMQP messages in several queues, you need to define the number of queue that you need and a sharding key. If you have a numerical ID, you can use the modulo X (X is the number of queue) as key and use the corresponding queue name. You can also use a hash or your data.
In your process, if the queue exists, add a task to it, or create it, and add it then. In any case, you can't have more than 1000 queues.