Search code examples
pythonsqldatabricksazure-databricks

Call one notebook from another notebook


I would like to call one notebook from another notebook in databricks.

lets say I have notebook_main and notebook_variable.

I want to run the notebook_variable from notebook_main and the out of the notebook vairiable I want to pass in the notebook_main.

For example:

In notebook_variable I have defiend one string variable

variable = "Select * From {table}"

{table} this has to become the dynamic variable in the second notebook.

Currently I have hardcode things in the notebook_main but I want to make it dynamic.

dbutils.widgets.text("table", "")
table = dbutils.widgets.get("table")
def update_changefeed(df, epochId):
    filtered_df = df.filter(col("_change_type").isin("insert", "update_postimage", "delete"))
    filtered_df.createOrReplaceGlobalTempView("test2")
    dfUpdates = sqlContext.sql("""
        SELECT * from  
        FROM global_temp.test2 """)

So with the above def Im trying to filter the df. Then create the global temp view out of it. Note that now the temp view name is hard coded and I need to pass the {table} name in it which will also need to be pass in the below select statement instead of global_temp.test2 it should be global_temp.{table}.

I have two main problem. How can I out put the notebook_variable result in this main nootebook and if the variables will be passed correctly.

My pseudo code:

dbutils.widgets.text("table", "")
table = dbutils.widgets.get("table")
query = out put of the notebook
def update_changefeed(df, epochId):
    filtered_df = df.filter(col("_change_type").isin("insert", "update_postimage", "delete"))
    filtered_df.createOrReplaceGlobalTempView(table)
    dfUpdates = sqlContext.sql(query)

Solution

  • Variable Notebook -

    table_name = "sample"
    dbutils.notebook.exit(table_name)
    

    You should pass the resulting variable on 'exit' method to get it in the main notebook.

    Main Notebook -

    table_name = dbutils.notebook.run("your_variable_notebook_path", 3600, {pass parameters if any in dict format})
    

    Now, use this table_name in your function

    (sample parameter - {"id":100001, "name":"Alpha"})