Terraform 403 error when creating function app and storage account with private endpoint

Question

I am getting a 403 forbidden when creating a function app that connects to its storage account via private endpoint inside a vnet. Storage account has firewall default action of 'Deny', and of course if I set it to 'Allow' it will work. I want this as 'Deny', however. Following this microsoft link if the function app and storage account are created in the same region with vnet, subnets, and private endpoints then it's supposed to work so I must be doing something wrong. I also tried changing the region for the storage account and it still resulted in a 403.

Error:

Error: web.AppsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="BadRequest" Message="There was a conflict. The remote server returned an error: (403) Forbidden." Details=[{"Message":"There was a conflict. The remote server returned an error: (403) Forbidden."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","ExtendedCode":"01020","Message":"There was a conflict. The remote server returned an error: (403) Forbidden.","MessageTemplate":"There was a conflict. {0}","Parameters":["The remote server returned an error: (403) Forbidden."]}}]

Here is my terraform code

resource "azurerm_function_app" "func" {
  name                       = "${var.func_basics.name}-func"
  location                   = var.func_basics.location
  resource_group_name        = var.func_basics.resource_group_name
  app_service_plan_id        = azurerm_app_service_plan.svc_plan.id
  storage_account_name       = azurerm_storage_account.func_sa.name
  storage_account_access_key = azurerm_storage_account.func_sa.primary_access_key
  version                    = var.runtime_version
  https_only                 = true
  depends_on = [
    azurerm_storage_account.func_sa,
    azurerm_app_service_plan.svc_plan,
    azurerm_application_insights.func_ai,
    azurerm_virtual_network.func_vnet
  ]

  app_settings = merge(var.app_settings, local.additional_app_settings)

}

resource "azurerm_app_service_plan" "svc_plan" {
  name                = "${var.func_basics.name}-func-plan"
  location            = var.func_basics.location
  resource_group_name = var.func_basics.resource_group_name
  kind                = "elastic"

  sku {
    tier = "ElasticPremium"
    size = "EP1"
  }
}

resource "azurerm_application_insights" "func_ai" {
  name                = "${var.func_basics.name}-func-appi"
  location            = var.func_basics.location
  resource_group_name = var.func_basics.resource_group_name
  application_type    = var.ai_app_type
}

resource "azurerm_storage_account" "func_sa" {
  name                      = "st${lower(replace(var.func_basics.name, "/[-_]*/", ""))}"
  resource_group_name       = var.func_basics.resource_group_name
  location                  = var.func_basics.location
  account_tier              = var.sa_settings.tier
  account_replication_type  = var.sa_settings.replication_type
  account_kind              = "StorageV2"
  enable_https_traffic_only = true
  min_tls_version           = "TLS1_2"
  depends_on = [
    azurerm_virtual_network.func_vnet
  ]

  network_rules {
    default_action = "Deny"
    virtual_network_subnet_ids = [azurerm_subnet.func_endpoint_subnet.id]
    bypass = [
      "Metrics",
      "Logging",
      "AzureServices"
    ]
  }
}

resource "azurerm_virtual_network" "func_vnet" {
  name                = "${var.func_basics.name}-func-vnet"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "func_service_subnet" {
  name                                          = "${var.func_basics.name}-func-svc-snet"
  resource_group_name                           = var.func_basics.resource_group_name
  virtual_network_name                          = azurerm_virtual_network.func_vnet.name
  address_prefixes                              = ["10.0.1.0/24"]
  enforce_private_link_service_network_policies = true

  service_endpoints = ["Microsoft.Storage"]

  delegation {
    name = "${var.func_basics.name}-func-del"

    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

resource "azurerm_subnet" "func_endpoint_subnet" {
  name                                           = "${var.func_basics.name}-func-end-snet"
  resource_group_name                            = var.func_basics.resource_group_name
  virtual_network_name                           = azurerm_virtual_network.func_vnet.name
  address_prefixes                               = ["10.0.2.0/24"]
  enforce_private_link_endpoint_network_policies = true
}

resource "azurerm_private_endpoint" "func_req_sa_blob_endpoint" {
  name                = "${var.func_basics.name}-func-req-sa-blob-end"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  subnet_id           = azurerm_subnet.func_endpoint_subnet.id

  private_service_connection {
    name                           = "${var.func_basics.name}-func-req-sa-blob-pscon"
    private_connection_resource_id = azurerm_storage_account.func_sa.id
    is_manual_connection           = false
    subresource_names              = ["blob"]
  }
}

resource "azurerm_private_endpoint" "func_req_sa_file_endpoint" {
  name                = "${var.func_basics.name}-func-req-sa-file-end"
  resource_group_name = var.func_basics.resource_group_name
  location            = var.func_basics.location
  subnet_id           = azurerm_subnet.func_endpoint_subnet.id

  private_service_connection {
    name                           = "${var.func_basics.name}-func-req-sa-file-pscon"
    private_connection_resource_id = azurerm_storage_account.func_sa.id
    is_manual_connection           = false
    subresource_names              = ["file"]
  }
}

resource "azurerm_app_service_virtual_network_swift_connection" "func_vnet_swift" {
  app_service_id = azurerm_function_app.func.id
  subnet_id      = azurerm_subnet.func_service_subnet.id
}

locals {
  additional_app_settings = {
    "APPINSIGHTS_INSTRUMENTATIONKEY" = azurerm_application_insights.func_ai.instrumentation_key
    "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING" = azurerm_storage_account.func_sa.primary_connection_string
    "AzureWebJobsStorage"                      = azurerm_storage_account.func_sa.primary_connection_string
    "WEBSITE_VNET_ROUTE_ALL"                   = "1"
    "WEBSITE_CONTENTOVERVNET"                  = "1"
    "WEBSITE_DNS_SERVER"                       = "168.63.129.16"
  }
}

Answer 1

It seems that it's a common error message when you create an Azure function where the storage account of the function is added to the Virtual Network, read here for more details.

To resolve it, you can use the local-exec Provisioner to invoke the az CLI command to deny the traffic after all of the provisions are finished.

az storage account update --name storage_account_name --resource-group reource_group_name --default-action 'Deny' --bypass 'AzureServices', 'Logging', 'Metrics'

Alternatively, you can separately configure the storage account network rules. You may need to allow your client's IP to access the storage account.

    resource "azurerm_storage_account_network_rules" "test" {
      resource_group_name  = var.resourceGroupName
      storage_account_name = azurerm_storage_account.func_sa.name
    
    
        default_action = "Deny"
       
        bypass = [
          "Metrics",
          "Logging",
          "AzureServices"
        ]
       ip_rules  = ["x.x.x.x"]
       
        depends_on = [
        azurerm_storage_account.func_sa,
        azurerm_app_service_plan.svc_plan,
        azurerm_application_insights.func_ai,
        azurerm_virtual_network.func_vnet,
        azurerm_function_app.func
      ]

}

In addition, there is a possible solution for this similar case on Github.