Search code examples
amazon-web-servicesdatabricksaws-databricks

Databricks AWS compute cluster location


I have hosted Databricks on top of AWS, but I cannot see any EC2 instance created for Databricks. Can anyone explain me, if I create databricks using aws in my VPC, will the computation be created outside my AWS VPC? If yes, where will the computation be created?


Solution

  • Except for Serverless options like Serverless SQL Warehouses, Databricks has an architecture that separates your Cloud environment (the Data Plane) from their cloud environment (the Control Plane). All of your compute (EC2 instances / VMs) and data are created within your Cloud environment. https://docs.databricks.com/getting-started/overview.html

    If you launch a Databricks Cluster in an AWS Databricks workspace, you can find it in the EC2 console. Make sure you are looking in the same AWS region that your Databricks workspace is deployed to. You can also use tags to filter to EC2 instances provisioned by Databricks in your AWS account, such as the Vendor=Databricks tag which is automatically added to all clusters:

    https://docs.databricks.com/administration-guide/account-settings/usage-detail-tags.html