Search code examples
azure-machine-learning-service

Run independent `PythonScriptStep` steps in parallel

In my pipeline multiple steps are independent and so I would like them to run in parallel based on input dependencies.

As the compute I use has multiple nodes I would have expected this to be the default.

For example:

enter image description here

All 3 upper steps should run in parallel, then both finetune steps in parallel as soon as their inputs are satisfied and the same for rgb_test.

Currently only 1 step runs at a time, the other are Queued.

Solution

  • It ended up being because of vCPU quota.

    After increasing the quota, parallel tasks can run at the same time as expected.