Databricks multi-task jobs - pass MLflow run_id from one task to next task

Question

I would like to create a databricks multi-task with following sequence:

• notebook task 1: train model with results logged to MLflow tracking server
• notebook task 2: use mlflow run_id from task 1 to register model in model registry

Is it possible to pass run_id from task 1 to task 2 and if so is there any documentation on how this could be done?

Answer 1

As of right now (it may change), it's impossible to pass results between jobs if you use multi-task job.

But you can call another notebook as a child job if you use notebook workflows and function dbutils.notebooks.run:

# notebook 1
... training code ...
dbutils.notebooks.run("notebook2", 300, {"run_id": run_id})