Search code examples
azure-machine-learning-service

How to organize one step after another in Azure Machine Learning Pipelines?

I have defined an Azure Machine Learning Pipeline with three steps:

e2e_steps=[etl_model_step, train_model_step, evaluate_model_step]
e2e_pipeline = Pipeline(workspace=ws, steps = e2e_steps)

The idea is to run the Pipeline in the given sequence:

  1. etl_model_step
  2. train_model_step
  3. evaluate_model_step

However, my experiment is failing because it is trying to execute evaluate_model_step before train_model_step: enter image description here

How do I enforce the sequence of execution?

Solution

  • azureml.pipeline.core.StepSequence lets you do exactly that.

    A StepSequence can be used to easily run steps in a specific order, without needing to specify data dependencies through the use of PipelineData.

    See the docs to read more.

    However, the preferable way to have steps run in order is stitching them together via PipelineData or OutputFileDatasetConfig. In your example, does the train_step depend on outputs from the etl step? If so, consider having that be the way that steps are run in sequence. For more info see this tutorial for more info