Search code examples
azuresizeazure-databricksazure-machine-learning-serviceazure-resource-graph

How to get list of compute instance size under Azure Machine Learning and Azure Databricks?


Goal here is to query a list of frequently used compute instance size under Azure Machine Learning and Azure Databricks using Azure Resource Graph Explorer from Azure Portal using Kusto query. From the documentation here, there is a list of resources can be queried but there isn't any compute under microsoft.machinelearningservices/(not classic studio) and Microsoft.Databricks/workspaces.

Below is what was tried, to get VM instance size but not showing what we have under Azure Machine Learning/Azure Databricks.

Resources
| project name, location, type, vmSize=tostring(properties.hardwareProfile.vmSize)
| where type =~ 'Microsoft.Compute/virtualMachines'
| order by name desc

Solution

  • Unfortunately, Azure Resource Graph Explorer doesn't provide any query to get any compute related information from both, Azure Machine Learning and Databricks.

    Though Azure Resource Graph Explorer supports join functionality, allowing for more advanced exploration of your Azure environment by enabling you to correlate between resources and their properties. But these services only applicable on few Azure resources like VM, storage account, Cosmos DB, SQL databases, Network Security Groups, public IP addresses, etc.

    Hence, there is no such Kusto query available in Azure Resource Graph Explorer which can list compute instance size of Machine Learning service and Databricks.

    Workarounds

    Machine Learning Service

    For machine learning service you can manage the compute instance directly from ML service by using Python SDK. Refer Python SDK azureml v1 to know more.

    Azure Databricks

    Cluster is the computational resource in Databricks. You can filter the cluster list from Databricks UI and manage the same. Features like cluster configuration, cluster cloning, access control, etc. are available which you can used based on your requirement. For more details, please check here.