azure data factory pipeline and activities Monitor

Question

I have about 120 pipeline with almost 400 activities all together and I would like to log them in our datalake storage system so we can report on the performance using powerBI. I came across How to get output parameter from Executed Pipeline in ADF? but it seems to me to work with a single pipeline, but I am wondering if I could get the whole pipeline in my ADF in one single call and the activities also.

Thnaks

Answer 1

Assuming the source in these pipelines varies which makes it difficult to apply the logic for monitoring.

One way is to store the logs individually for each pipeline by running some queries with pipeline parameters. Refer Option 2 in this tutorial.

Although, the best feasible and appropriate way to monitor ADF pipelines and activities is to use the Azure Data Factory Analytics.

This solution provides you a summary of overall health of your Data Factory, with options to drill into details and to troubleshoot unexpected behavior patterns. With rich, out of the box views you can get insights into key processing including:

• At a glance summary of data factory pipeline, activity and trigger runs
• Ability to drill into data factory activity runs by type
• Summary of data factory top pipeline, activity errors

1. Go to Azure Marketplace, choose Analytics filter, and search for Azure Data Factory Analytics (Preview)

2. Select Create and then create or select the Log Analytics Workspace.

Installing this solution creates a default set of views inside the workbooks section of the chosen Log Analytics workspace. As a result, the following metrics become enabled:

• ADF Runs - 1) Pipeline Runs by Data Factory
• ADF Runs - 2) Activity Runs by Data Factory
• ADF Runs - 3) Trigger Runs by Data Factory
• ADF Errors - 1) Top 10 Pipeline Errors by Data Factory
• ADF Errors - 2) Top 10 Activity Runs by Data Factory
• ADF Errors - 3) Top 10 Trigger Errors by Data Factory
• ADF Statistics - 1) Activity Runs by Type
• ADF Statistics - 2) Trigger Runs by Type
• ADF Statistics - 3) Max Pipeline Runs Duration

You can visualize the preceding metrics, look at the queries behind these metrics, edit the queries, create alerts, and take other actions.