Terraform 403 Error for Restricted Storage Account

Question

I tried to create a Storage Container within a Storage Account, while the Storage Account shouldn't be accessed by the public. I mostly followed this tutorial to use a private endpoint: https://gmusumeci.medium.com/using-private-endpoint-in-azure-storage-account-with-terraform-49b4734ada34. I created it successfully. But now, when I want to make any changes I will get a 403 Error:

Error: retrieving Container "tfstate" (Account "Account \"tfstate2hnq5\" (IsEdgeZone false / ZoneName \"\" / Subdomain Type \"blob\" / DomainSuffix \"core.windows.net\")"): executing request: unexpected status 403 (403 This request is not authorized to perform this operation.) with AuthorizationFailure: This request is not authorized to perform this operation.

I have all the necessary access rights, I also tried to use a Service Principal and also give it all the rights you can have. Still, it doesn't work. Did I overlook something here? I also can't access the storage container on Azure portal.

My main.tf:

resource "random_string" "resource_code" {
  length  = 5
  special = false
  upper   = false
}

resource "azurerm_resource_group" "tfstate" {
  name     = "tfstate"
  location = "West Europe"
}

resource "azurerm_virtual_network" "this" {
  name                = "network-vnet"
  address_space       = [var.network-vnet-cidr]
  resource_group_name = azurerm_resource_group.tfstate.name
  location            = azurerm_resource_group.tfstate.location
}

resource "azurerm_subnet" "endpoint-subnet" {
  name                                      = "endpoint-subnet"
  address_prefixes                          = [var.endpoint-subnet-cidr]
  virtual_network_name                      = azurerm_virtual_network.this.name
  resource_group_name                       = azurerm_resource_group.tfstate.name
  private_endpoint_network_policies = "Enabled"
  service_endpoints = ["Microsoft.Storage", "Metrics"]
}

resource "azurerm_private_dns_zone" "dns-zone" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.tfstate.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "network_link" {
  name                  = "network_link"
  resource_group_name   = azurerm_resource_group.tfstate.name
  private_dns_zone_name = azurerm_private_dns_zone.dns-zone.name
  virtual_network_id    = azurerm_virtual_network.this.id
}

resource "azurerm_private_endpoint" "endpoint" {
  name                = "tfstate_pe"
  resource_group_name = azurerm_resource_group.tfstate.name
  location            = azurerm_resource_group.tfstate.location
  subnet_id           = azurerm_subnet.endpoint-subnet.id

  private_service_connection {
    name                           = "tfstate_psc"
    private_connection_resource_id = azurerm_storage_account.tfstate.id
    is_manual_connection           = false
    subresource_names              = ["blob"]
  }
}

resource "azurerm_private_dns_a_record" "storage_account" {
  name                = azurerm_storage_account.tfstate.name
  zone_name           = azurerm_private_dns_zone.dns-zone.name
  resource_group_name = azurerm_resource_group.tfstate.name
  ttl                 = 300
  records             = [azurerm_private_endpoint.endpoint.private_service_connection.0.private_ip_address]
}

resource "azurerm_storage_account" "tfstate" {
  name                            = "tfstate${random_string.resource_code.result}"
  resource_group_name             = azurerm_resource_group.tfstate.name
  location                        = azurerm_resource_group.tfstate.location
  account_tier                    = "Standard"
  account_replication_type        = "LRS"
  account_kind                    = "StorageV2"
  default_to_oauth_authentication = true
  enable_https_traffic_only       = true
  min_tls_version                 = "TLS1_2"

  blob_properties {
    versioning_enabled            = true
    change_feed_enabled           = true
    change_feed_retention_in_days = 90
    last_access_time_enabled      = true
  delete_retention_policy {
    days = 90
  }

  container_delete_retention_policy {
    days = 30
  }

}
  }


resource "azurerm_storage_account_network_rules" "rules" {
  storage_account_id = azurerm_storage_account.tfstate.id
  default_action     = "Deny"
  bypass             = ["AzureServices"]
  virtual_network_subnet_ids = [azurerm_subnet.endpoint-subnet.id]
}

resource "azurerm_storage_container" "tfstate" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.tfstate.name
  container_access_type = "private"
}

resource "azurerm_storage_account_network_rules" "rules" {
  storage_account_id = azurerm_storage_account.tfstate.id
  default_action     = "Deny"
  bypass             = ["AzureServices"]
  virtual_network_subnet_ids = [azurerm_subnet.endpoint-subnet.id]
}

resource "azurerm_storage_container" "tfstate" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.tfstate.name
  container_access_type = "private"
}

Answer 1

Container creation using terraform in and storage account with private end point.

The issue you're facing is due to the firewall network rule not allowing to create the container inside it. For this to work we need to create a new rule to allow the IP to get access to create a container or choose an IP that allow by firewall.

Terraform configuration:

provider "azurerm" {
  features {}
}

resource "random_string" "resource_code" {
  length  = 5
  special = false
  upper   = false
}

resource "azurerm_resource_group" "tfstate" {
  name     = "tfstate"
  location = "West Europe"
}

resource "azurerm_virtual_network" "this" {
  name                = "network-vnet"
  address_space       = ["10.0.0.0/16"]
  resource_group_name = azurerm_resource_group.tfstate.name
  location            = azurerm_resource_group.tfstate.location
}

resource "azurerm_subnet" "endpoint-subnet" {
  name                                      = "endpoint-subnet"
  address_prefixes                          = ["10.0.1.0/24"]
  virtual_network_name                      = azurerm_virtual_network.this.name
  resource_group_name                       = azurerm_resource_group.tfstate.name
  private_endpoint_network_policies = "Enabled"
  service_endpoints = ["Microsoft.Storage"]
}

resource "azurerm_private_dns_zone" "dns-zone" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.tfstate.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "network_link" {
  name                  = "network_link"
  resource_group_name   = azurerm_resource_group.tfstate.name
  private_dns_zone_name = azurerm_private_dns_zone.dns-zone.name
  virtual_network_id    = azurerm_virtual_network.this.id
}

resource "azurerm_private_endpoint" "endpoint" {
  name                = "tfstate_pe"
  resource_group_name = azurerm_resource_group.tfstate.name
  location            = azurerm_resource_group.tfstate.location
  subnet_id           = azurerm_subnet.endpoint-subnet.id

  private_service_connection {
    name                           = "tfstate_psc"
    private_connection_resource_id = azurerm_storage_account.tfstate.id
    is_manual_connection           = false
    subresource_names              = ["blob"]
  }
}

resource "azurerm_private_dns_a_record" "storage_account" {
  name                = azurerm_storage_account.tfstate.name
  zone_name           = azurerm_private_dns_zone.dns-zone.name
  resource_group_name = azurerm_resource_group.tfstate.name
  ttl                 = 300
  records             = [azurerm_private_endpoint.endpoint.private_service_connection.0.private_ip_address]
}

resource "azurerm_storage_account" "tfstate" {
  name                            = "tfstate${random_string.resource_code.result}"
  resource_group_name             = azurerm_resource_group.tfstate.name
  location                        = azurerm_resource_group.tfstate.location
  account_tier                    = "Standard"
  account_replication_type        = "LRS"
  account_kind                    = "StorageV2"
  default_to_oauth_authentication = true
  enable_https_traffic_only       = true
  min_tls_version                 = "TLS1_2"

  blob_properties {
    versioning_enabled            = true
    change_feed_enabled           = true
    change_feed_retention_in_days = 90
    last_access_time_enabled      = true
  delete_retention_policy {
    days = 90
  }

  container_delete_retention_policy {
    days = 30
  }

}
  }


resource "azurerm_storage_account_network_rules" "rules" {
  storage_account_id = azurerm_storage_account.tfstate.id
  default_action     = "Deny"
  ip_rules = ["125.19.127.99"]
  bypass             = ["AzureServices"]
  virtual_network_subnet_ids = [azurerm_subnet.endpoint-subnet.id]
}

resource "azurerm_storage_container" "tfstate" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.tfstate.name
  container_access_type = "private"

  depends_on = [ azurerm_storage_account_network_rules.rules ]
}

Depolyment succeded:

Reference:

https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints