Search code examples
azureazure-data-factorydatabricksazure-databrickscost-management

Azure Cost Management - track costs associated with Databricks job

I have an Azure Data Factory pipeline that runs multiple Databricks Notebooks using job clusters. I want to track down the cost of those job clusters in the Cost Management panel.

What I'm interested in is not only the Databricks cost, but also the underlying VM cost, but only for that specific set of jobs.

I know that I can filter by Tag and then by jobid, but that needs a lot of manual work and of course, the jobid changes between pipeline runs.

Is there any way to tag or filter it in a more automated fashion? Maybe by service principal used to run those jobs?

Solution

  • I ended up with dynamically tagging the job clusters with Data Factory's pipeline ID during pipeline run. The ADF activity used to run Databricks notebook has an option to add cluster tags. I can then reference the pipeline ID in Cost Managent to filter information I need.