Search code examples
azurenetwork-programmingterraformazure-keyvaultazure-application-gateway

Error ERR_SSL_UNRECOGNIZED_NAME_ALERT from Azure Application Gateway with SSL Certificate in Key Vault of Different Subscription

I've setup an application gateway and key vault using Terraform. All seems to work with the application gateway if I store the certificate locally, but if I set it to use a certificate from the key vault, then browse to the site, I get the error ERR_SSL_UNRECOGNIZED_NAME_ALERT, implying that the listener has been disabled. I've seen others have reported similar issues (e.g. Azure Application Gateway, how to link to SSL Cert on a different Azure Subscription?); though none of the answers there work for me / some are now out of date.

  • Though the Azure Portal UI doesn't support key vault in a different subscription, Azure itself does (related docs); so I was able to set the certificate via TerraForm.
    • Side note: I also manually uploaded the certificate directly to the AppGW using the portal to prove that all worked when the certificate was local; it did.
    • Side note 2: I then used PowerShell to set the value back to using the Key Vault in case there was some Terraform issue; but using PowerShell gave the same result (i.e. successfully updated the config, but caused the ERR_SSL_UNRECOGNIZED_NAME_ALERT exception to reappear). Set-AzApplicationGatewaySslCertificate -ApplicationGateway $appGw -Name 'wildcard-example-com' -KeyVaultSecretId 'https://myKeyVault.vault.azure.net/secrets/wildcard-example-com/' -Verbose | Set-AzApplicationGateway -Verbose
  • All resources are in the same region (UK South).
  • My application gateway is connected to a VNet
    • This association is configured by passing the subnet id to the gateway_ip_configuration block.
    • The VNet is connected to our VHub.
    • There is a routing rule to ensure traffic to private IP ranges is directed via our FW (where we host rules for internal traffic), whilst traffic to public ranges goes directly out to the internet per docs.
    • There are no rules to block traffic between the AppGW and the FW (no NSGs configured on the relevant subnets, no rules that would affect this on the FW; FW logs don't show traffic between AppGW and KeyVault (Denied or otherwise), though we have full logging enabled).
    • The subnet has the Microsoft.KeyVault service endpoint associated
    • Our internal DNS servers are associated with the VNet / these correctly resolve the key vault's internal IP address (i.e. showing that conditional forwarders & private dns zones for the key vault are correctly configured)
  • My key vault is connected to a different VNet
    • The key vault is connected to this vnet via private endpoint
    • The key vault has Disable public access and Allow trusted Microsoft services to bypass this firewall set
    • This vnet is also peered with the VHub, with all traffic being routed via the FW (i.e meaning connections to/from other vnets will go via the FW as well as public egress; there is no public ingress as that's blocked by the key vault's network configuration as mentioned above).
  • The AppGW has a managed identity associated
  • This managed identity is configured in the key vault's access policy, with get permissions for secrets (in fact, I've currently granted every permission available, just to discount security policies as being the potential cause). Note: I'm using vault access policy; not RBAC.
  • The path used in the app gateway's ssl certificate's setting is correct - i.e. no typos, points to the correct key vault, points to the /secrets/ path, and doesn't include version information.

Is there anything I've missed, or are there any debugging steps I can take to get more info on why AppGW is failing to load this certificate from the KeyVault?

Solution

  • I had a call with MS Support where they were able to resolve my issue.

    The private dns zone used by key vault, privatelink.vaultcore.azure.net had Virtual Network Links configured with the VNet in which my domain controllers sit; these domain controllers have conditional dns forwarders for the various zones including vaultcore.azure.net forwarding requests to 168.63.129.16. This means that when our DCs are used for DNS, they resolve resources to the internal IP of the private endpoints.

    However, though the VNet to which the AppGW is correctly configured to use our DNS servers, and thus should be using the internal IP to connect to the Key Vault securely, that doesn't seem to have happenned.

    By adding another Virtual Network Link to our privatelink.vaultcore.azure.net zone, targetting the App Gateway's VNet, all began to work.

    Note: Immediately after making the above change things still failed; I found that running Set-AzApplicationGatewaySslCertificate -ApplicationGateway $appGw -Name 'wildcard-example-com' -KeyVaultSecretId 'https://myKeyVault.vault.azure.net:443/secrets/wildcard-example-com/' -Verbose | Set-AzApplicationGateway -Verbose caused the AppGW to retry pulling the cert, thus triggering it to take advantage of the now fixed issue. (I'm not sure if it's important, but each time I ran the above command I'd alternate between including the :443 port info to ensure the string was different, so a change would be detected.

    Hope that's helpful to others in future.