Azure quota limit issue when deploying Databricks via Terraform

Question

I am trying to deploy a databricks workspace in Azure and create a Single Node cluster. To do that, I use the following Terraform main.tf file:

   terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.55.0"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "1.0.0"
    }
  }
}

# Configure Azure provider
provider "azurerm" {
  features {}
}

# Configure Databricks provider
provider "databricks" {
  host = azurerm_databricks_workspace.databricks_workspace.workspace_url
}

# Create resource group
resource "azurerm_resource_group" "resource_group" {
  name     = var.resource_group_name
  location = var.resource_group_location
}

# Create Databricks workspace
resource "azurerm_databricks_workspace" "databricks_workspace" {
  location            = azurerm_resource_group.resource_group.location
  name                = "databricks-test-001"
  resource_group_name = azurerm_resource_group.resource_group.name
  sku                 = "standard"

  depends_on = [
    azurerm_resource_group.resource_group
  ]
}

# Create cluster
data "databricks_node_type" "smallest" {
  local_disk = true

  depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true

  depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}

resource "databricks_cluster" "single_node" {
  cluster_name            = "Single Node"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 10

  spark_conf = {
    # Single-node
    "spark.databricks.cluster.profile" : "singleNode"
    "spark.master" : "local[*]"
  }

  custom_tags = {
    "ResourceClass" = "SingleNode"
  }

  depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}

# Create Notebook
resource "databricks_notebook" "notebook" {
  content_base64 = base64encode("print('Welcome to Databricks-Labs notebook')")
  path           = "/Shared/Demo/demo_example_notebook"
  language       = "PYTHON"

  depends_on = [
    databricks_cluster.single_node,
    azurerm_databricks_workspace.databricks_workspace
  ]
}

When I apply the terraform plan, I get the following error:

Required: 24, (Minimum) New Limit Required: 24. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22a9f6a84e-aa76-4493-ad46-7335d8bc7ea5%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardNCADSA100v4Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:24,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardNCADSA100v4Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests databricks_error_message:Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved StandardNCADSA100v4Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: westus, Current Limit: 0, Current Usage: 0, Additional Required: 24, (Minimum) New Limit Required: 24. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22a9f6a84e-aa76-4493-ad46-7335d8bc7ea5%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22westus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22StandardNCADSA100v4Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:24,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22StandardNCADSA100v4Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests]. Please see https://docs.databricks.com/dev-tools/api/latest/clusters.html#clusterclusterstate for more details

I get that the quota limit needs to be increased and I have submitted a request, which was denied. I have also tried to do the deployment in different Azure regions, which was also unsuccessful for the same reason. What I don't get is why if I go to the created Databricks workspace and try to create a Single Node cluster from there it works with no issues at all. To me this means that the quota limit might not be the actual issue. I would appreciate any suggestions about what might be the issue.

Answer 1

I have checked with the following code and tried creating a Single Node cluster.

Main.tf:

resource "azurerm_databricks_workspace" "databricks_workspace" {
  location            = data.azurerm_resource_group.example.location
  name                = "databricks-test-001"
  resource_group_name = data.azurerm_resource_group.example.name
  sku                 = "standard"

 
}

data "databricks_current_user" "me" {
  depends_on = [azurerm_databricks_workspace.databricks_workspace]
}
data "databricks_spark_version" "latest" {
  depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}
data "databricks_spark_version" "latest_lts" {
 long_term_support = true
 depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}
data "databricks_node_type" "smallest" {   
 local_disk = true
 depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}


resource "databricks_cluster" "single_node" {
  cluster_name            = "Single Node"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 10

  spark_conf = {
    # Single-node
    "spark.databricks.cluster.profile" : "singleNode"
    "spark.master" : "local[*]"
  }

  custom_tags = {
    "ResourceClass" = "SingleNode"
  }

  depends_on = [
    azurerm_databricks_workspace.databricks_workspace
  ]
}

Upon terraform apply:

It got executed successfully:

The issue mainly occurs due to azure quota limitation for regions. As you said even after changing the region , the issue prevails, the issue must be due to misconfiguration of azure subscription on logging in with incorrect azure creadetials .

Make sure to set the correct subscription after logging in with az login.

az account set --subscription "xxxx"

main.tf

provider "databricks" {
  host = azurerm_databricks_workspace.databricks_workspace.workspace_url
}



provider "azuread" {
   subscription_id = "xxx
      tenant_id              = "xxx"
}



data "azuread_client_config" "current" {
  //tenant_id = "xxxx"
}


data "azurerm_client_config" "current" {
  //tenant_id = "xxx"
 // subscription_id = "8ae0844f-xxx"
}


data azurerm_subscription "current"{
//  subscription_id = "xxx"
}


data "azurerm_storage_account" "example" {
name = "remoxxxx"
  resource_group_name = data.azurerm_resource_group.example.name
}


terraform {
  backend "azurerm" {
    resource_group_name  = "vxxx"
    storage_account_name = "remotestatekavstr233"
    container_name       = "terraform"
    key                  = "terraform.tfstate"
  }
}

If the subscription is confirmed to set properly , then the subscription might have reached its limit.

Please check Troubleshoot QuotaExceeded error code - Azure | Microsoft Learn