Passing failed notebook Error to outside of "Execute Pipeline" in Azure Data Factory (ADF)

Question

So, I have an "Execute Pipeline" in ADF and inside of it I have multiples notebook's that are writing queries to tables.

I also a databricks notebook with a class for sending e-mails (that notebook receives dynamically, using widget's, the error, and this error is what is going to be send as the body of the e-mail)

Well, currently I have this working: if a notebook fails, the entire "Execute Pipeline" fails. Now, I what I want to do is if a notebook fails, it goes to the e-mail notebook with the error and it sends an e-mail.

I tried this:

The parameter that I am giving the notebook error is:

It does work, but it clutters the pipeline to much!

Ideally, I would like something like this, but I am having trouble passing the error of the failed notebook from inside the "Execute Pipeline" to the outside.

Does anybody knows how can I do this? Or have a different way? I was also thinking about a Try-Catch inside the pipeline but I am not sure it would work :/

Thank you!! :D

Answer 1

In order to pass value from one pipeline to activity of the other pipeline, you can use pipeline return value option in set variable activity.

• Click the Wait on completion in the execute pipeline activity.

• In the pipeline, which is invoked through execute pipeline activity, add a set variable activity and create a new variable and define the variable. Here, I created a variable called return_val and notebook output is given as value for that variable.

• In the main pipeline after execute activity, you can add any activity and you can pass the value from child pipeline.

@activity('Execute Pipeline1').output.pipelineReturnValue.return_val

Here return_val is the variable name that I used in the child pipeline.