I would like to call one notebook from another notebook in databricks.
lets say I have notebook_main and notebook_variable.
I want to run the notebook_variable from notebook_main and the out of the notebook vairiable I want to pass in the notebook_main.
For example:
In notebook_variable I have defiend one string variable
variable = "Select * From {table}"
{table}
this has to become the dynamic variable in the second notebook.
Currently I have hardcode things in the notebook_main but I want to make it dynamic.
dbutils.widgets.text("table", "")
table = dbutils.widgets.get("table")
def update_changefeed(df, epochId):
filtered_df = df.filter(col("_change_type").isin("insert", "update_postimage", "delete"))
filtered_df.createOrReplaceGlobalTempView("test2")
dfUpdates = sqlContext.sql("""
SELECT * from
FROM global_temp.test2 """)
So with the above def
Im trying to filter the df
. Then create the global temp view out of it. Note that now the temp view name is hard coded and I need to pass the {table} name in it which will also need to be pass in the below select
statement instead of global_temp.test2
it should be global_temp.{table}
.
I have two main problem. How can I out put the notebook_variable result in this main nootebook and if the variables will be passed correctly.
My pseudo code:
dbutils.widgets.text("table", "")
table = dbutils.widgets.get("table")
query = out put of the notebook
def update_changefeed(df, epochId):
filtered_df = df.filter(col("_change_type").isin("insert", "update_postimage", "delete"))
filtered_df.createOrReplaceGlobalTempView(table)
dfUpdates = sqlContext.sql(query)
Variable Notebook -
table_name = "sample"
dbutils.notebook.exit(table_name)
You should pass the resulting variable on 'exit' method to get it in the main notebook.
Main Notebook -
table_name = dbutils.notebook.run("your_variable_notebook_path", 3600, {pass parameters if any in dict format})
Now, use this table_name
in your function
(sample parameter - {"id":100001, "name":"Alpha"})