Is it possible to disable Databricks Managed Storage Account public access?
We have a problem with public access on the managed storage account from Databricks in Azure.
Databricks has a default managed resource group, with a managed storage account that Databricks uses for the standard hive metastore (we do not use this metastore, since we are using unity catalog).
This storage account has public access enabled, but our Azure policies do not allow public access on storage accounts.
We are deploying using Terraform and as far as we can see there are no options to close the public access for this storage account.
As a sidenote: we have Databricks deployed with VNET injection.
One hacky solution is to always run an az cli command after our Databricks deployment to close the access, but we would rather not do this (edit: this is also not possibly due to deny assignment on the st).
Does anyone know if this access can be configured?
May 2024: it's now possible to make this storage account private: https://learn.microsoft.com/en-us/azure/databricks/security/network/storage/firewall-support
No, it's not possible to do this right now, and really you can't make this change because of how managed resource group is configured. And the public access is required for Databricks to work as many things, such as, logs, models, etc. are stored on DBFS Root (managed storage account) and needs to be accessed by the Azure Databricks control plane.
But if it comes to the data security, here are the relevant information. The storage account has the deny policy that prevents from making changes & accessing the data for anyone except the Databricks application. So even it has public access, you can't generate SAS, use storage account key, etc.
- How to enable "Use for Hive table metadata" in "AWS Glue Data Catalog settings" using Terraform?
- Unexpected JWT alg received, expected RS256, got: HS256
- terraform tags for list variable not working
- Terraform 13, Validate variable base on the value of another
- How do I reference a resource created from different module in my current module?
- Terraform - GCP - Import project IAM member
- Creating an Azure Linux VM with Ubuntu 20.04 with Terraform
- Terraform: Merge two tuples/lists to create a map
- Terraform doesn't allow to use variable when creating resource. How to go around?
- Authenticating using the Azure CLI is only supported as a User (not a Service Principal)
- Terraform outputs 'Error: Variables not allowed' when doing a plan
- Issue with Azure key vault and Azure storage account deployment
- temporary_name_for_rotation must be specified when changing any of the following properties: pod_subnet_id
- Why is TF build failing with "Error refreshing state: HTTP remote state endpoint requires auth"?
- With Gitlab, in the CI/CD pipeline, how can I dynamically create and save SSH key pairs as CI/CD project variables?
- Conditionally add element to existing map
- Parse HCL block into Golang map
- Run an AWS ECS task
- Terraform Error: Error locking state: Error acquiring the state lock: 2 errors occurred:
- Terraform: Error: retrieving static website properties for Storage Account (Subscription: *** : context deadline exceeded
- How to add additional security groups to EKS cluster?
- How to use a blue/green deployment style with CodeDeploy and Auto Scaling Groups that allows for repetitive deployments in Terraform?
- Self referenced Security group inbound rule in AWS RDS terraform?
- How to run this terraform nested for loop?
- Error: Failure when executing a Terraform Resource creation request
- Conditionally add property to object
- Registering a private endpoint in Azure private DNS automatically using Terraform
- How to deal with "Error: Duplicate object key" in Terraform
- Unable to use VARs from a tfvars file in Terraform
- Terraform AWS Provider: SecretsManager can't apply because version was deleted