databricks notebook to read databricks job statistics and write it on a snowflake table

Question

I have several jobs scheduled in databricks, out of which I am interested in reading statistics of two specific jobs. I need to write a databricks notebook code to write the job statistics (jobName, startTime, endTime and status) into a snowflake table.

Answer 1

We can use following python code to get the details from databricks job api.

Note : tested code here

from pyspark.sql.types import IntegerType
from pyspark.sql.types import *
from pyspark.sql import Row
import base64
import requests
import json

databricks_instance ="<databricks-instances>"

url_list = f"{databricks_instance}/api/2.0/jobs/runs/get?run_id=39347"

headers = {
  'Authorization': 'Bearer <databricks-access-token>',
  'Content-Type': 'application/json'
}

response = requests.request("GET", url_list, headers=headers).json()
print(response)
print(response['job_id'])
print(response['start_time'])
print(response['end_time'])