I have set up a private worker pool and I had expected that the queued time for a build would go down. Previously queue time (when only one build is queued) was about 1 minute. I had assumed because I was using shared machines inside GCP to do the build. I therefore expected that a private worker pool would have no queue since I would be the only one building anything. I was surprised to see it took about 1 minute also. I then thought that perhaps the first build would have to spin up a VM and that's why it took so long so I carried out a second build after the first had finished but that also had a queue time of about 1 minute. I don't understand what is going on, 1 minute is quite a long time.
When you use Cloud Build shared pool, you use machine provisioned by Google, up and running and paid by Google. Therefore, when you have a build to run, you pick one active machine in the shared pool and you run your build on it.
With private pool, it's different. The machine are still managed by Google, but the pool is private, dedicated to you. Therefore, Google won't keep VM up and running (and use CPU/Memory) if you run nothing on them (because you pay only when a job is running). So, Google stop the VM.
When you run a job on Cloud Build, a VM is started and your job can start. Similar to Compute Engine, it takes about 1 minutes to provision a VM and to start it.
That being said, your requirement could be a nice feature request: keep warm a number of VMs to prevent this on demand provisioning. Of course, it won't be free, but it will be faster!
You can open a feature request here