I'm setting up an ingress for the application resides in the AKS. But ran into a problem on binding the certificate to the ingress.
As you can see below, I am trying to reference ingress-cert from the KV and use it in the ingress through SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: {{ include "secretProvider.name" . }}
spec:
provider: azure
secretObjects:
- secretName: ingress-tls-csi
type: kubernetes.io/tls
data:
- objectName: ingress-cert
key: tls.key
- objectName: ingress-cert
key: tls.crt
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: {{ .Values.keyVault.identity }}
keyvaultName: {{ .Values.keyVault.name }}
objects: |
array:
- |
objectName: ingress-cert
objectType: secret
tenantId: {{ .Values.keyVault.tenant }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "ingress.name" . }}
annotations:
kubernetes.io/ingress.class: azure/application-gateway
kubernetes.io/ingress.allow-http: "false"
appgw.ingress.kubernetes.io/override-frontend-port: "443"
spec:
tls:
- hosts:
- {{ .Values.ingress.host }}
secretName: ingress-tls-csi
# Property `rules` is omitted
It's working fine when accessing other secrets from pods through env but for the ingress this is the output on describing it:
Name: my-ingress
Namespace: my-namespace
Address: x.x.x.x
Default backend: default-http-backend:80
TLS:
ingress-tls-csi terminates my.example.com
Rules:
Host Path Backends
---- ---- --------
Annotations: appgw.ingress.kubernetes.io/override-frontend-port: 443
kubernetes.io/ingress.allow-http: false
kubernetes.io/ingress.class: azure/application-gateway
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SecretNotFound 3m3s (x2 over 3m10s) azure/application-gateway Unable to find the secret associated to secretId: [my-namespace/ingress-tls-csi]
I've set up the KV integration by following the Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster documentation.
But upon following the Set up Secrets Store CSI Driver to enable NGINX Ingress Controller with TLS as a hint on how to implement the same with AGIC. I noticed that the certificate is added as a secret inside the AKS which is then referenced inside the ingress with secretName: ingress-tls-csi.
kubectl get secret -n $NAMESPACE
NAME TYPE DATA AGE
ingress-tls-csi kubernetes.io/tls 2 1m34s
I assume that ingress can't reference the secret directly from SecretProviderClass as the example in the documentation need to use the ingress-tls-csi as a secret object which I assumed (again) created by ingress-nginx chart.
My question is how can I implement the same as the ingress-nginx example with AGIC?
Additional information:
kubectl command. The reason I need to use the one from KV is the AKS will also be used by other people deploying under the same domain but different namespace and I think it's a bad idea to give direct access to certificate's private key.As I couldn't find a way to integrate the Ingress with the Azure Key Vault, I've implemented a workaround with GitHub Actions to retrieve the certificate and add it to the AKS. Because most of them are bash commands, the workaround isn't exclusive to GitHub Actions.
name: Assign Cerfiticate from KV to AKS
on:
workflow_dispatch:
jobs:
build-push:
name: Assign Cerfiticate from KV to AKS
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
# Azure Authentication
- uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- name: Authenticate and set context to AKS cluster
run: az aks get-credentials --name "my-aks-cluster" --resource-group "my-rg" --admin
- uses: azure/setup-helm@v3
with:
token: ${{ secrets.PAT }}
# Retrieve certificate and private key from Azure Key Vault
- name: Download certificate and private key in PKCS#12 format
run: |
az keyvault secret download --name my-certificate \
--vault-name my-kv \
--encoding base64 \
-f certificate.pfx
- name: Extract private key in RSA format
run: |
openssl pkcs12 -in certificate.pfx -nocerts -nodes -passin pass: | openssl rsa -out private.key
- name: Extract certificate
run: |
openssl pkcs12 -in certificate.pfx -clcerts -nokeys -passin pass: -out certificate.crt
- name: Deploy Kubernetes Configuration
run: |
helm upgrade --install release-name chart-name \
-n my-namespace \
--set-file mychart.ingress.key=private.key \
--set-file mychart.ingress.certificate=certificate.crt
The brief explanation for the above:
.pfx format.crt and .keyhelm install/upgrade using --set-fileHere is the secret and ingress configuration:
apiVersion: v1
kind: Secret
metadata:
name: ingress-tls
type: kubernetes.io/tls
data:
tls.crt: {{ .Values.ingress.certificate | b64enc }}
tls.key: {{ .Values.ingress.key | b64enc }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "ingress.name" . }}
annotations:
kubernetes.io/ingress.class: azure/application-gateway
kubernetes.io/ingress.allow-http: "false"
appgw.ingress.kubernetes.io/override-frontend-port: "443"
spec:
tls:
- hosts:
- {{ .Values.ingress.host }}
secretName: ingress-tls
# Property `rules` is omitted
No additional modification to the SecretProviderClass is required.
I hope this is just a workaround because it'd be nicer if the Ingress can directly integrate with Azure Key Vault.