Synapse pipeline Logs don't appear in log analytics

Question

I have several pipeline in Synapse that preprocess data.

Data in the wrong format goes in a 'df_to_consult'.

I am trying to be notified/alerted everytime new data go into this df.

What I did is to create a diagnostic setting where destination details is Log Analytics workspace.

For the categories I selected : Integration Pipeline Runs, Integration Activity Runs, Integration Trigger Runs.

I launch my pipelines with debug button or trigger.

But when I go on Logs I see no logs and I don't know why.

What can I do to see logs related to fail or executed pipelines ?

Answer 1

I have used an workaround using Logic Apps to complete this action

Synapse: I have used a Fail Activiy to initaite pipeline failure. and using the Pipeline runId as an example.

Logic App Workflow:

You need to onthe managed id in Logic apps and add it in synapse access control. Reference

Step 1: To create the pipeline run using HTTP action. Using the POST method to create the synapse pipeline run.

https://<Synapse workspace name>azuresynapse.net/pipelines/<Pipeline name>/createRun?api-version=2020-12-01

Step 2: Initiate a Array variable using a initialize variable.

Step 3: Take a until loop first use the below RestApi to get the above pipeline run details.Using the GET method.

https://synapsedileep.dev.azuresynapse.net/pipelineruns/@{body('HTTP').runId}?api-version=2020-12-01

then check the status of the pipeline run in every iteration. if it is failed thne send the mail using IF condition true activity. Use the same pipeline run status in the until loop condition. That means this loop will run until the pipeline status is either Failed or succeeded.

Output:

You can use the below code view:

{  
"definition": {  
"$schema": "[https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#"](https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#%22 "https://schema.management.azure.com/providers/microsoft.logic/schemas/2016-06-01/workflowdefinition.json#%22"),  
"actions": {  
"HTTP": {  
"inputs": {  
"authentication": {  
"audience": "[https://dev.azuresynapse.net/"](https://dev.azuresynapse.net/%22 "https://dev.azuresynapse.net/%22"),  
"type": "ManagedServiceIdentity"  
},  
"method": "POST",  
"uri": "[https://synapsedileep.dev.azuresynapse.net/pipelines/Pipeline1/createRun?api-version=2020-12-01"](https://synapsedileep.dev.azuresynapse.net/pipelines/Pipeline1/createRun?api-version=2020-12-01%22 "https://synapsedileep.dev.azuresynapse.net/pipelines/pipeline1/createrun?api-version=2020-12-01%22")  
},  
"runAfter": {},  
"type": "Http"  
},  
"Initialize_variable": {  
"inputs": {  
"variables": [  
{  
"name": "var",  
"type": "string"  
}  
]  
},  
"runAfter": {  
"HTTP": [  
"Succeeded"  
]  
},  
"type": "InitializeVariable"  
},  
"Initialize_variable_1": {  
"inputs": {  
"variables": [  
{  
"name": "var2",  
"type": "array",  
"value": [  
"Failed",  
"Succeeded"  
]  
}  
]  
},  
"runAfter": {  
"Initialize_variable": [  
"Succeeded"  
]  
},  
"type": "InitializeVariable"  
},  
"Until": {  
"actions": {  
"Condition": {  
"actions": {  
"Send_an_email_(V2)": {  
"inputs": {  
"body": {  
"Body": "<p>@{outputs('HTTP_1')?['statusCode']} : @{body('HTTP_1').status}</p>",  
"Importance": "Normal",  
"Subject": "IMPORTANT",  
"To": "v-dthumula@microsoft.com"  
},  
"host": {  
"connection": {  
"name": "@parameters('$connections')['office365']['connectionId']"  
}  
},  
"method": "post",  
"path": "/v2/Mail"  
},  
"type": "ApiConnection"  
}  
},  
"else": {  
"actions": {}  
},  
"expression": {  
"and": [  
{  
"equals": [  
"@body('HTTP_1').status",  
"Failed"  
]  
}  
]  
},  
"runAfter": {  
"HTTP_1": [  
"Succeeded"  
]  
},  
"type": "If"  
},  
"HTTP_1": {  
"inputs": {  
"authentication": {  
"audience": "[https://dev.azuresynapse.net/"](https://dev.azuresynapse.net/%22 "https://dev.azuresynapse.net/%22"),  
"type": "ManagedServiceIdentity"  
},  
"method": "GET",  
"uri": "[https://synapsedileep.dev.azuresynapse.net/pipelineruns/@{body('HTTP').runId}?api-version=2020-12-01"](https://synapsedileep.dev.azuresynapse.net/pipelineruns/@%7Bbody(%27HTTP%27).runId%7D?api-version=2020-12-01%22 "https://synapsedileep.dev.azuresynapse.net/pipelineruns/@%7bbody(%27http%27).runid%7d?api-version=2020-12-01%22")  
},  
"type": "Http"  
}  
},  
"expression": "@contains(variables('var2'),body('HTTP_1').status)",  
"limit": {  
"count": 60,  
"timeout": "PT1H"  
},  
"runAfter": {  
"Initialize_variable_1": [  
"Succeeded"  
]  
},  
"type": "Until"  
}  
},  
"contentVersion": "1.0.0.0",  
"outputs": {},  
"parameters": {  
"$connections": {  
"defaultValue": {},  
"type": "Object"  
}  
},  
"triggers": {  
"When_a_HTTP_request_is_received": {  
"kind": "Http",  
"type": "Request"  
}  
}  
},  
"parameters": {  
"$connections": {  
"value": {  
"office365": {  
"connectionId": "/subscriptions/b83c1ed3/resourceGroups/thumula/providers/Microsoft.Web/connections/office365",  
"connectionName": "office365",  
"id": "/subscriptions/b83c1ed3/providers/Microsoft.Web/locations/eastus/managedApis/office365"  
}  
}  
}  
}  
}

Note: you should run your synapse pipelin from the logic apps to get the alerts: