azure databricks: exceeding approved Total Regional Cores quota
I have a "Pay as you Go" Azure subscription.
Currently I have no active resources in this subscription
(all of the resources involved in prior projects have been deleted).
I have created a new Databricks premium resource and proceeded to execute the provided Delta Live Tables example by choosing to create a new DLT pipeline from the example code.
This attempt have failed with the following error:
DataPlaneException: [DLT ERROR CODE: CLUSTER_LAUNCH_FAILURE.CLIENT_ERROR]
Failed to launch pipeline cluster 0928-153235-l27bxsvs with termination code
INVALID_ARGUMENT and termination type CLIENT_ERROR: Attempt to launch cluster with
invalid arguments. databricks_error_message: Operation could not be completed as it
results in exceeding approved Total Regional Cores quota. Additional details - Dep...
I have checked my Azure Compute quotas - there is plenty of availability as I am not currently using any other Compute resources.
Is there a solution to this problem?
The reason is that quotas are used for different resource groups, subscriptions, accounts, and scopes. For instance, the number of cores for a particular region may be restricted by the subscription you have. An error claiming the quota has been exceeded appears if you try to install a virtual machine with more cores than are allowed.
To check your subscription's Usage + Quotas.
Choose a subscription >> Settings >> Usage + Quotas. Select Microsoft.Compute using the filter.
Verify total regional vCPU consumption here: If the consumption is at capacity, you must choose Request Increase to raise the region's core limit. From here only you have New Quota Request Option to raise Increase quota limit
