My company is in the process of migrating all our pipelines over to Databricks from AzureML, and I have been tasked with refactoring one of our existing pipelines made with azureml-sdk (using functions such as PipelineData, PythonScriptStep etc.), and converting it into a dbx pipeline which uses a deployment.yml file.
I have found this "Deployment file reference" on dbx documentation page, and I think it's quite adequate compared to some of AzureML's documentation. However, if I had an example project to compliment that page, it would help me greatly to put it into practice.
Is there any repos/sources which gives an example of building a dbx pipeline which uses .py-files instead of notebooks?
However, if I had an example project to compliment that page, it would help me greatly to put it into practice.
Please take a look at the Quickstart doc which generates a sample project and walks you through it step by step.
If you're looking for more profound and in-depth example with orientation towards MLOps practices, take a look at the following session - MLOps on Databricks: A How-To Guide. It also links to an example repo with dbx.