Missing cluster cert causes Add-AzServiceFabricClusterCertificate to fail: Object reference not set to an instance of an object

Question

I'm fairly new to Service Fabric, so I'm not sure if this is an issue with the cmdlet or if this is a miss on my part. I am using Az.ServiceFabric module version 2.0.2 and the Az module version 3.8.0.

I am trying to use the Add-AzServiceFabricClusterCertificate cmdlet to add a secondary certificate that I've already created in my Azure KeyVault to my cluster. When I run the cmdlet, it fails with this error (running with Debug gave me more stack detail):

DEBUG: AzureQoSEvent: CommandName - Add-AzServiceFabricClusterCertificate; IsSuccess - False; Duration -
00:00:07.3059582;; Exception - System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Azure.Commands.ServiceFabric.Commands.ServiceFabricClusterCmdlet.GetClusterType(Cluster
clusterResource)
   at Microsoft.Azure.Commands.ServiceFabric.Commands.AddAzureRmServiceFabricClusterCertificate.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();

Looking at the code for this cmdlet, I noticed that it's probably failing because the cluster resource that gets passed into GetClusterType does not have its Certificate member, so it fails when it tries to check the Certificate.Thumbprint and Certificate.ThumbprintSecondary:

        internal ClusterType GetClusterType(Cluster clusterResource)
        {
            if (string.IsNullOrWhiteSpace(clusterResource.Certificate.Thumbprint) &&
                string.IsNullOrWhiteSpace(clusterResource.Certificate.ThumbprintSecondary))
            {
                return ClusterType.Unsecure;
            }
            else
            {
                return ClusterType.Secure;
            }
        }

The cluster that gets passed into GetClusterType is retrieved in the same manner as in the Get-AzServiceFabricCluster cmdlet, so when I run that cmdlet for the cluster that I'm trying to add the certificate to, I noticed that my Certificate field is empty in the response. I'm guessing that's what's causing the NullRef exception. Here's that relevant snippet:

          AzureActiveDirectory :
              TenantId : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
              ClusterApplication : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
              ClientApplication : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          Certificate :
          CertificateCommonNames : Microsoft.Azure.Management.ServiceFabric.Models.ServerCertificateCommonNames
          ClientCertificateCommonNames :
          ClientCertificateThumbprints :

I'm wondering if it's expected that the Certificate field would be empty when I'm using the Get-AzServiceFabricCluster cmdlet and if that is indeed the cause of my Add-AzServiceFabricClusterCertificate cmdlet failing. When I look at the cluster's Security blade in Azure Portal, I do see the Primary Cluster Certificate with which I originally created the cluster, and this is the cert that I use when deploying and doing other cluster operations. However, I did notice that the cert thumbprint field is empty when viewing the certificate from the portal. I would expect to see this certificate when using Get-AzServiceFabricCluster, but it comes up empty. Is this certificate missing from my Get-AzServiceFabricCluster cmdlet possible to fix through the portal or with another cmdlet?

Answer 1

It looks like your cluster is configured to find certificates by common name, rather than thumbprint. I'm guessing this based on the fact your portal doesn't show a thumbprint against the certificate, in addition to the snippet you have posted.

If this is the case, there's no need to update your cluster configuration with a new certificate when the old certificate has expired - Instead you need to install the certificate only into your VMSS vault. Once you add the new certificate to the VMSS, Service Fabric will automatically use the later expiring certificate.

You must always ensure you have at least one valid certificate installed on your VMSS with the common name configured in your cluster.

PS to upload certificate KV and install onto VMSS:

$subscriptionId  = "sub-id"
$vmssResourceGroupName     = "vmss-rg-name"
$vmssName                  = "vmss-name"
$vaultName                 = "kv-name"
$primaryCertName           = "kv-cert-name"
$certFilePath              = "...\.pfx"
$certPassword              = ConvertTo-SecureString -String "password" -AsPlainText -Force

# Sign in to your Azure account and select your subscription
Login-AzAccount -SubscriptionId $subscriptionId

# Update primary certificate within the Key Vault
$primary = Import-AzKeyVaultCertificate `
    -VaultName $vaultName `
    -Name $primaryCertName `
    -FilePath $certFilePath `
    -Password $certPassword

$certConfig = New-AzVmssVaultCertificateConfig -CertificateUrl $primary.SecretId -CertificateStore "My"

# Get VM scale set 
$vmss = Get-AzVmss -ResourceGroupName $vmssResourceGroupName -VMScaleSetName $vmssName

# Add new certificate version
$vmss.VirtualMachineProfile.OsProfile.Secrets[0].VaultCertificates.Add($certConfig)

# Update the VM scale set 
Update-AzVmss -ResourceGroupName $vmssResourceGroupName -Verbose `
    -Name $vmssName -VirtualMachineScaleSet $vmss

For more info, I wrote a blog post on switching from thumbprint to common name.

The official docs are also a good reference.