How to abort a notebook in DataFactory

Question

I am trying to write a query to a table in data factory. I already have functions and query and it works fine. Now I want to abort the notebook as soon as the code enters the except part. I have this right now:

try:
    df = spark.sql(query)
    df.write.format(formato).mode(method).saveAsTable(table)
    except Exception as e:
        msg = "An error occured while writing the query to the table: {}".format(e)
        dbutils.notebook.exit(msg)

It does actually show a message on the notebook and stop it from continuing. BUT, on data factory it does show that the run was successfully , but I wanted to show as "Failed" .

Does anyone can help me?

Thank you so much!! :)

Answer 1

I tried the same with your code and my Notebook is also gave me Success.

Here, you are handling the error and passing a message to the notebook using dbutils.notebook.exit(msg).

Notebook won't fail when it exited with a message but the databricks notebook execution will be aborted.

If you want to get the reason(Message) causing the abort of the databricks notebook in the ADF pipeline, use the below expression after the Success of the notebook activity.

@activity('Notebook1').output.runOutput

(OR)

If you want Notebook activity to fail, you need to generate the error and abort the notebook like below sample.

try:
    df = spark.sql(query)
    df.write.format(formato).mode(method).saveAsTable(table)
except Exception as e:
    msg = "An error occured while writing the query to the table: {}".format(e)
    logging.error(msg)
    raise

Now, to get the error message in ADF, use the below expression after the Failure of the Notebook activity.

@activity('Notebook1').output.runError