Search code examples
kubernetesazure-akskubernetes-secrets

Secret creation with SecretProviderClass not working as aspected


EDIT: It was a config error, I was setting wrong kv name :/

As said in title I'm facing an issue with secret creation using SecretProviderClass.

I've created my aks and my kv (and filled it) on azure. then I'll proceed to follow those steps using a user-assigned managed identity

but no secret resource get created and pods got stuck on creation with mount failure.

those are the steps I followed

az extension add --name aks-preview
az extension update --name aks-preview
az aks enable-addons --addons azure-keyvault-secrets-provider -g $RESOURCE_GROUP -n $AKS_CLUSTER 
az aks update -g $RESOURCE_GROUP -n $AKS_CLUSTER --enable-managed-identity --disable-secret-rotation
$AKS_ID = (az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query identityProfile.kubeletidentity.clientId -o tsv)
az keyvault set-policy -n $AZUREKEYVAULT --secret-permissions get --spn $AKS_ID

the SecretProviderClass manifest I'm using

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  secretObjects:
  - secretName: akvsecrets
    type: Opaque
    data:
    - objectName: AzureSignalRConnectionString 
      key: AzureSignalRConnectionString
    - objectName: BlobStorageConnectionString 
      key: BlobStorageConnectionString
    - objectName: SqlRegistryConnectionString 
      key: SqlRegistryConnectionString
    - objectName: TokenSymmetricKey 
      key: TokenSymmetricKey
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: XXX # VMSS UserAssignedIdentity
    keyvaultName: "sampleaks001" # the name of the KeyVault
    objects:  |
      array:
        - |
          objectName: AzureSignalRConnectionString
          objectType: secret
        - |
          objectName: BlobStorageConnectionString
          objectType: secret
        - |
          objectName: SqlRegistryConnectionString
          objectType: secret
        - |
          objectName: TokenSymmetricKey
          objectType: secret
    resourceGroup: sample # [REQUIRED for version < 0.0.4] the resource group of the KeyVault
    subscriptionId: XXXX # [REQUIRED for version < 0.0.4] the subscription ID of the KeyVault
    tenantId: XXX # the tenant ID of the KeyVault  

and the deploy manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trm-api-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: trm-api-test
  template:
    metadata:
      labels:
        app: trm-api-test
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: trm-api-test
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: AzureSignalRConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: AzureSignalRConnectionString
        - name: TokenSymmetricKey
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: TokenSymmetricKey
        - name: BlobStorageConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: BlobStorageConnectionString
        - name: SqlRegistryConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: SqlRegistryConnectionString
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kvname"
---
apiVersion: v1
kind: Service
metadata:
  name: trm-api-service-test
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: trm-api-test
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP

I'm sure I'm missing something, but can't understand what. Thanks in advance!


Solution

  • you are using the clientId, but it should be the objectId form the kubelet identity:

    export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
    export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
    az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID
    

    This is a working SecretProviderClass i am using (adjusted to your config):

    apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: azure-kvname
    spec:
      provider: azure
      secretObjects:
      - data:
        - objectName: AzureSignalRConnectionString 
          key: AzureSignalRConnectionString
        - objectName: BlobStorageConnectionString 
          key: BlobStorageConnectionString
        - objectName: SqlRegistryConnectionString 
          key: SqlRegistryConnectionString
        - objectName: TokenSymmetricKey 
          key: TokenSymmetricKey
        secretName: akvsecrets
        type: Opaque
      parameters:
        usePodIdentity: "false"
        useVMManagedIdentity: "true"
        userAssignedIdentityID: XXX # Kubelet Client Id ( Nodepool Managed Idendity )
        keyvaultName: "sampleaks001" # the name of the KeyVault
        tenantId: XXX # the tenant ID of the KeyVault  
        objects:  |
          array:
            - |
              objectName: AzureSignalRConnectionString
              objectAlias: AzureSignalRConnectionString
              objectType: secret
            - |
              objectName: BlobStorageConnectionString
              objectAlias: BlobStorageConnectionString
              objectType: secret
            - |
              objectName: SqlRegistryConnectionString
              objectAlias: SqlRegistryConnectionString
              objectType: secret
            - |
              objectName: TokenSymmetricKey
              objectAlias: TokenSymmetricKey
              objectType: secret
    

    You can also check documentation here as you will find better examples as on the Azure Docs.