Accessing Azure Key Vault from Azure Batch

Question

I am trying to access secrets in my Azure Key Vault from VMs running in my Azure Batch node pool.

However, I keep running into the exception:

Exception Message: Tried 1 certificate(s). Access token could not be acquired.

Exception for cert #1 with thumbprint MY-THUMBPRINT: Keyset does not exist

So far I have been following the instructions outlined here: https://learn.microsoft.com/en-us/azure/key-vault/service-to-service-authentication

The article outlines the scenario of Azure Batch, and indicates I should use a Service Principal. I would like to ensure no secrets or keys are in version control, so I am using the first method of a certificate in a local keystore to sign into Azure AD.

Running all of the below locally as an executable works fine, but fails when run on an Azure Batch pool node.

My steps so far to do the above are:

1. Create a service principal and associated certificate in keyvault: az ad sp create-for-rbac --name myserviceprincipal --create-cert --cert mycertname --keyvault mykeyvaultname. Keep the service principal app id and tenant id for use in the AzureServicesAuthConnectionString.

2. Create key vault access policy for the created service principal (done in the azure portal UI).

3. Download the created certificate in PFX/PEM format (done in the Azure Portal UI).

4. Ensuring a PFX password on the certificate (I am doing this as uploading the cert to azure batch in step 6 requires an associated password): https://coombes.nz/blog/azure-keyvault-export-certificate/

# Replace these variables with your own values
$vaultName = "YOUR-KEYVAULT-NAME"
$certificateName = "YOUR-CERTIFICATE-NAME"
$pfxPath = [Environment]::GetFolderPath("Desktop") + "\$certificateName.pfx"
$password = "YOUR-CERTIFICATE-PASSWORD"

$pfxSecret = Get-AzureKeyVaultSecret -VaultName $vaultName -Name $certificateName
$pfxUnprotectedBytes = [Convert]::FromBase64String($pfxSecret.SecretValueText)
$pfx = New-Object Security.Cryptography.X509Certificates.X509Certificate2
$pfx.Import($pfxUnprotectedBytes, $null, [Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
$pfxProtectedBytes = $pfx.Export([Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12, $password)
[IO.File]::WriteAllBytes($pfxPath, $pfxProtectedBytes)

5. Installing the certificate locally into my computer's LocalMachine store (for local testing).

6. Uploading the certificate into Azure Batch (using the Azure Portal UI upload).

7. Associating the certificate with the appropriate Node Pool and rebooting the nodes (using the Azure Portal UI for now).

My application package running on Azure Batch is a simple console executable. The AzureServicesAuthConnectionString is set to RunAs=App;AppId={AppId};TenantId={TenantId};CertificateThumbprint={Thumbprint};CertificateStoreLocation={LocalMachine} and the remaining key vault code to retreive the secret looks like:

Environment.SetEnvironmentVariable("AzureServicesAuthConnectionString", "RunAs=App;AppId=<MY-APP-ID>;TenantId=<MY-TENANT>;CertificateThumbprint=<MY-THUMBPRINT>;CertificateStoreLocation=LocalMachine");

AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider(Environment.GetEnvironmentVariable("AzureServicesAuthConnectionString"));
KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = await keyVaultClient.GetSecretAsync("<MY-SECRET>").ConfigureAwait(false);
var message = secret.Value;
Console.WriteLine(message);

Things work fine locally, but fail on the remote node. I am able to RDP into the Azure Batch node and see that the certificate has been installed for the local machine.

I am wondering how to resolve my error or if my above steps are wrong in some way?

Answer 1

In order to access the Certificate, it must be associated and installed for the 'Current User.' It may be that LocalMachine does not have the appropriate level of permissions?

On Azure Batch, ensure that the uploaded certificate is associated with:

Store Name: 'My' Store Location: 'CurrentUser'

This post was helpful: https://github.com/nabhishek/customactivity_sample/tree/linkedservice

As was this post: X509Certificate - Keyset does not exist

The connection string in the C# exe looks like:

Environment.SetEnvironmentVariable("AzureServicesAuthConnectionString",
                    "RunAs=App;" +
                    "AppId=<the app id of my active directory app registration> ;" +
                    "TenantId=<my subscription tenant id>;" +
                    "CertificateThumbprint=<the thumbprint of my cert>;" +
                    "CertificateStoreLocation=CurrentUser");

                AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider(Environment.GetEnvironmentVariable("AzureServicesAuthConnectionString"));
                KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
                var secret = await keyVaultClient.GetSecretAsync("https://<my vault name>.vault.azure.net/secrets/<secret name>/<secret id>")
                        .ConfigureAwait(false);
                message = secret.Value;
                Console.WriteLine(message);