Search code examples
pysparkdatabricksaws-databricksdatabricks-workflows

How to extract failure messages from databricks job runs?

I am working on creating a dataframe which ultimately will feed into a tableau report. The data is supposed to be extracted from the databricks job run for failed jobs and along witht he reason of the failure. I was able to extract the failed jobs by using the /api/2.1/jobs/runs/get?run_id. But I am struggling with finding a way to extract the error messages from the notebook url (the api doesn't have)

for example : I need to read the below AnalysisException:'and the message here'

Does anyone have a suggestion?

Thanks in advance.

I tried to use different api in databricks and tried to read from notebook url.

Solution

  • Use this api /api/2.1/jobs/runs/get-output. This is the link to its documentation: https://docs.databricks.com/api/workspace/jobs/getrunoutput. If the job has failed, there's going to be an error field in the API response.