Terraform Kubernetes Provider - cannot load Kubernetes client config

Question

In my terraform project, I have my own module which creates an AKS cluster with RBAC enabled. This module has an output aks_public_fqdn which holds the FQDN of the cluster.

module "aks" {
  source = "./aks"

  environment        = local.environment
  region             = local.region
  instance_id        = local.workload_id
  application        = local.workload_name
  resource_group     = local.rg_name
  kubernetes_version = local.kubernetes_version

}

Once the cluster is up and running, I would like to create some K8S resources through Terraform with the Kubernetes provider.

To authenticate I am using the following code snippet:

provider "kubernetes" {
  host                   = "https://${module.aks.aks_public_fqdn}"
  insecure = true

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "kubelogin"
    args = [
      "get-token",
      "--environment",
      "AzurePublicCloud",
      "--server-id",
      "3333333-3333333-3333333",
      "--login",
      "azurecli"
    ]
  }
}

When I try to run terraform apply I get:

Error: Provider configuration: cannot load Kubernetes client config
invalid configuration: default cluster has no server defined

The only entry I have in my kubeconfig file is the context for my local kind cluster. What is missing here?

Am I hit by the following github issue? Provider Issue

Answer 1

The kubernetes provider configuration contains an output from the module declared as aks. Prior to version 2.4.0 of the Kubernetes provider it was possible to simultaneously manage a Kubernetes cluster backing infrastructure and the cluster itself at initial provisioning (or subsequent Delete/Create) by configuring the provider with either resource attributes, or with data attributes. At version 2.4.0 of the Kubernetes provider, the new experimental Kubernetes provider with the latest Terraform SDK and Kubernetes Go SDK bindings (and consequently Kubernetes API) stabilized and replaced the former Kubernetes provider (ergo why some resources are marked v1 and v2 as the provider still contained some of the legacy code for backwards support, and only the minor version of the provider was iterated and not the major version according to semantic versioning rules). With this new provider and its use of the modern Kubernetes API it became no longer possible for this simultaneous management at initial provisioning. Therefore the error in the question is observed as Terraform's Kubernetes provider is attempting to configure with a non-existent cluster instead of charting a dependency hierarchy that would imply the provider is dependent upon the aks module.

With all this in mind it becomes clear there are two workarounds. The first is to downgrade the Kubernetes provider to the last version of the old provider:

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.3.2"
    }
  }
}

The other solution would be to -target the aks module first to manually enforce the dependency and populate the outputs, and then subsequently manage the entire Terraform config:

terraform plan -target=module.aks