ADO Pipeline Environment Kubernetes On-Prem Resource Connection failing with x509: certificate signed by unknown authority

Question

I am trying to setup a multi-stage ADO pipeline using ADO pipeline Environment feature.

Stage 1: Builds the Spring-boot based Java Micro-service using Maven.

Stage 2: Deploys the above using Helm 3. The HelmDeploy@0 task uses Environment which has a Resource called tools-dev (a kubernetes namespace) where I want this service to be deployed using Helm chart.

It fails at the last step with this error:

/usr/local/bin/helm upgrade --install --values /azp/agent/_work/14/a/values.yaml --wait --set ENV=dev --set-file appProperties=/azp/agent/_work/14/a/properties.yaml --history-max 2 --stderrthreshold 3 java-rest-template k8s-common-helm/rest-template-helm-demo

Error: Kubernetes cluster unreachable: Get "https://rancher.msvcprd.windstream.com/k8s/clusters/c-gkffz/version?timeout=32s": x509: certificate signed by unknown authority
##[error]Error: Kubernetes cluster unreachable: Get "https://rancher.msvcprd.windstream.com/k8s/clusters/c-gkffz/version?timeout=32s": x509: certificate signed by unknown authority**

Finishing: Helm Deploy

I created the Kubernetes resource in the Environment using the kubectl commands specified in the settings section.

Deploy stage pipeline excerpt:

- stage: Deploy
  displayName: kubernetes deployment
  dependsOn: Build
  condition: succeeded('Build')
  jobs:
  - deployment: deploy
    pool: $(POOL_NAME)
    displayName: Deploy
    environment: dev-az-s-central-k8s2.tools-dev
    strategy:
      runOnce:
        deploy: 
          steps:
          - bash: |
              helm repo add \
                k8s-common-helm \
                http://nexus.windstream.com/repository/k8s-helm/
              helm repo update
            displayName: 'Add and Update Helm repo'
            failOnStderr: false
          - task: HelmDeploy@0
            inputs:
              command: 'upgrade'
              releaseName: '$(RELEASE_NAME)'
              chartName: '$(HELM_CHART_NAME)'
              valueFile: '$(Build.ArtifactStagingDirectory)/values.yaml'
              arguments: '--set ENV=$(ENV) --set-file appProperties=$(Build.ArtifactStagingDirectory)/properties.yaml  --history-max 2 --stderrthreshold 3'
            displayName: 'Helm Deploy'

Environment Settings: Name: dev-az-s-central-k8s2 Resource: tools-dev (Note: this is an on-prem k8s cluster that I am trying to connect to).

Can you please let me know what additional configuration is required to resolve this x509 certificate issue?

Answer 1

Check this documentation:

The issue is that your local Kubernetes config file must have the correct credentials.

When you create a cluster on GKE, it will give you credentials, including SSL certificates and certificate authorities. These need to be stored in a Kubernetes config file (Default: ~/.kube/config) so that kubectl and helm can access them.

Also, check answer in case Helm 3: x509 error when connecting to local Kubernetes

Helm looks for kubeconfig at this path $HOME/.kube/config.

Please run this command

microk8s.kubectl config view --raw > $HOME/.kube/config

This will save the config at required path in your directory and shall work