Machine learning in Azure: How do I publish a pipeline to the workspace once I've already built it in Python using the SDK?

Question

I don't know where else to ask this question so would appreciate any help or feedback. I've been reading the SDK documentation for azure machine learning service (in particular azureml.core). There's a class called Pipeline that has methdods validate() and publish(). Here are the docs for this:

https://learn.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipeline.pipeline?view=azure-ml-py

When I call validate(), everything validates and I call publish but it seems to only create an API endpoint in the workspace, it doesn't register my pipeline under Pipelines and there's obviously nothing in the designer.

My question: I want to publish my pipeline so I just have to launch from the workspace with one click. I've built it already using the SDK (Python code). I don't want to work with an API. Is there any way to do this or would I have to rebuild the entire pipeline using the designer (drag and drop)?

Answer 1

Totally empathize with your confusion. Our team has been working with Azure ML pipelines for quite some time but PublishedPipelines still confused me initially because:

• what the SDK calls a PublishedPipeline is called as a Pipeline Endpoint in the Studio UI, and
• it is semi-related to Dataset and Model's .register() method, but fundamentally different.

TL;DR: all Pipeline.publish() does is create an endpoint that you can use to:

• schedule and version Pipelines, and
• re-run the pipeline from other services via a REST API call (e.g. via Azure Data Factory).

You can see PublishedPipelines in the Studio UI in two places:

• Pipelines page :: Pipeline Endpoints tab
• Endpoints page :: Pipeline Endpoints tab