DevOps CI/CD pipelines broken after Kubernetes upgrade to v1.22

Question

Present state

In v1.22 Kubernetes dropped support for v1beta1 API. That made our release pipeline crash and we are not sure how to fix it.

We use build pipelines to build .NET Core applications and deploy them to the Azure Container Registry. Then there are release pipelines that use helm to upgrade them in the cluster from that ACR. This is how it looks exactly.

Build pipeline:

1. .NET download, restore, build, test, publish
2. Docker task v0: Build task
3. Docker task v0: Push to the ACR task
4. Artifact publish to Azure Pipelines

Release pipeline:

1. Helm tool installer: Install helm v3.2.4 (check for latest version of Helm unchecked) and install newest Kubectl (Check for latest version checked)
2. Bash task:

az acr login --name <acrname>
az acr helm repo add --name <acrname>

3. Helm upgrade task:
   • chart name <acrname>/<chartname>
   • version empty
   • release name `

After the upgrade to Kubernetes v1.22 we are getting the following error in Release step 3.:

Error: UPGRADE FAILED: unable to recognize "": no matches for kind "Ingress" in version "extensions/v1beta1".

What I've already tried

Error is pretty obvious and from Helm compatibility table it states clearly that I need to upgrade the release pipelines to use at least Helm v3.7.x. Unfortunately in this version OCI functionality (about this shortly) is still in experimental phase so at least v3.8.x has to be used.

Bumping helm version to v3.8.0

That makes release step 3. report:

Error: looks like "https://<acrname>.azurecr.io/helm/v1/repo" is not a valid chart repository or cannot be reached: error unmarshaling JSON: while decoding JSON: json: unknown field "acrMetadata"

After reading Microsoft tutorial on how to live with helm and ACR I learned that az acr helm commands use helm v2 so are deprecated and OCI artifacts should be used.

Switching to OCI part 1

After reading that I changed release step 2. to a one-liner:

helm registry login <acrname>.azurecr.io --username <username> --password <password>

That now gives me Login Succeeded in release step 2. but release step 3. fails with

Error: failed to download "<acrname>/<reponame>".

Switching to OCI part 2

I thought that the helm task is incompatible or something with the new approach so I removed release step 3. and decided to make it from the command line in step 2. So now step 2. looks like this:

helm registry login <acrname>.azurecr.io  --username <username> --password <password>
helm upgrade --install --wait -n <namespace> <deploymentName> oci://<acrname>.azurecr.io/<reponame> --version latest --values ./values.yaml

Unfortunately, that still gives me:

Error: failed to download "oci://<acrname>.azurecr.io/<reponame>" at version "latest"

Helm pull, export, upgrade instead of just upgrade

The next try was to split the help upgrade into separately helm pull, helm export and then helm upgrade but

helm pull oci://<acrname>.azurecr.io/<reponame> --version latest

gives me:

Error: manifest does not contain minimum number of descriptors (2), descriptors found: 0

Changing docker build and docker push tasks to v2

I also tried changing the docker tasks in the build pipelines to v2. But that didn't change anything at all.

Answer 1

Have you tried changing the Ingress object's apiVersion to networking.k8s.io/v1beta1 or networking.k8s.io/v1? Support for Ingress in the extensions/v1beta1 API version is dropped in k8s 1.22.

Our ingress.yaml file in our helm chart looks something like this to support multiple k8s versions. You can ignore the AWS-specific annotations since you're using Azure. Our chart has a global value of ingress.enablePathType because at the time of writing the yaml file, AWS Load Balancer did not support pathType and so we set the value to false.

{{- if .Values.global.ingress.enabled -}}
{{- $useV1Ingress := and (.Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress") .Values.global.ingress.enablePathType -}}
{{- if $useV1Ingress -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: example-ingress
  labels:
    {{- include "my-chart.labels" . | nindent 4 }}
  annotations:
    {{- if .Values.global.ingress.group.enabled }}
    alb.ingress.kubernetes.io/group.name: {{ required "ingress.group.name is required when ingress.group.enabled is true" .Values.global.ingress.group.name }}
    {{- end }}
    {{- with .Values.global.ingress.annotations }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
    # Add these tags to the AWS Application Load Balancer
    alb.ingress.kubernetes.io/tags: k8s.namespace/{{ .Release.Namespace }}={{ .Release.Namespace }}
spec:
  rules:
    - host: {{ include "my-chart.applicationOneServerUrl" . | quote }}
      http:
        paths:
          {{- if $useV1Ingress }}
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ $applicationOneServiceName }}
                port:
                  name: http-grails
          {{- else }}
          - path: /*
            backend:
              serviceName: {{ $applicationOneServiceName }}
              servicePort: http-grails
          {{- end }}
    - host: {{ include "my-chart.applicationTwoServerUrl" . | quote }}
      http:
        paths:
          {{- if $useV1Ingress }}
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ .Values.global.applicationTwo.serviceName }}
                port:
                  name: http-grails
          {{- else }}
          - path: /*
            backend:
              serviceName: {{ .Values.global.applicationTwo.serviceName }}
              servicePort: http-grails
          {{- end }}
{{- end }}