Search code examples
azuredatabricksazure-databricks

How to terminate databricks cluster from notebook?


I want to terminate azure databricks cluster from notebook level. Is there any possibility?


Solution

  • As suggested by Ganesh Chandrasekaran in comments. You can choose Job cluster for your requirement.

    Job clusters terminate when your job ends, reducing resource usage and cost. The cluster is not terminated when idle but terminates only after all tasks using it have completed. If a shared job cluster fails or is terminated before all tasks have finished, a new cluster is created.

    A cluster scoped to a single task is created and started when the task starts and terminates when the task is completed. In production, Databricks recommends using new shared or task scoped clusters so that each job or task runs in a fully isolated environment.

    Refer - https://learn.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobs#choose-the-correct-cluster-type-for-your-job