Search code examples
azure-data-factoryazure-synapse

Easily access notebook output run in Synapse pipeline

I developed a synapse notebook which does data quality checks, visualizes null values and a couple of other charts I need to monitor data quality.

I created a pipeline in Synapse to run the notebook automatically every day. The way I can check the notebook after the run is to go in the Pipeline Runs, click on the Pipeline, click on Activity Name and then visualize the notebook output.

Is there a better way to check the notebook daily? Like a single bookmark for example instead of checking every single activity run with multiple clicks?

Or would it be possible to send the html output of the notebook by mail?

Thank you for your kind help.

Solution

  • would it be possible to send the html output of the notebook by mail?

    it is possible to send the html output of the notebook by mail.

    • To achieve that you have to store your html code into a variable and the pass that variable to mssparkutils.notebook.exit() as shown below. enter image description here
    • Then run the notebook in pipeline. it will exit the variable as exit value. enter image description here
    • Then store this exit value in a variable using @activity('Notebook 1').output.status.Output.result.exitValue enter image description here
    • Then to send an email you need to create a logic app to create it you can follow this document. but little change is there between HTTP request and send email you need to add 2 more things you need to initialize and set the variable. -- Http request settings: enter image description here -- Initialize variable enter image description here -- Set variable enter image description here -- Send an email v2 settings enter image description here

    Output:

    enter image description here