At my job, we recently converted some pipelines to be "all-in-one." So when a merge to main happens, the pipeline has multiple stages for building --> deploy to lower environments --> deploy to production. The production deployment requires manual approval. As a result, builds will often sit at this stage as we only deploy on a certain cadence.
As part of our trigger in the YAML file, we have batch: true. However, I believe this is causing all new merges to NOT automatically kick off the pipeline because we typically have prior runs that are sitting on the production deployment stages waiting for approval, so they technically haven't "completed" yet.
We really only need to batch based on the build stage, not the pipeline entirely. I found this thread from 2019 detailing this same issue. A response from a Microsoft rep says "We do have per-stage batching on the backlog and plan to work on it in the next few months" - it's been ~5 years since this post. I'm finding nothing on per-stage batching....does anyone know if this is possible?
According to the Note in Batching CI runs document,
If the pipeline has multiple jobs and stages, then the first run should still reach a terminal state by completing or skipping all its jobs and stages before the second run can start. For this reason, you must exercise caution when using this feature in a pipeline with multiple stages or approvals. If you wish to batch your builds in such cases, it is recommended that you split your CI/CD process into two pipelines - one for build (with batching) and one for deployments.
It seems the per-stage batching feature has not been released yet.
In this case, you can split your CI/CD process into two pipelines - one for build (with batching) and one for deployments. Then you can use the pipeline resource triggers in the deployment pipeline to make sure that the deployment pipeline runs after the build pipeline completes.
You can also request the feature for Azure DevOps here and vote on it. Engineers usually prioritize features that are urgent or have more votes.