Add secret to freshly created Azure AKS using Terraform Kubernetes provider fails

Question

I am creating a kubernetes cluster with the Azure Terraform provider and trying to add a secret to it. The cluster gets created fine but I am getting errors with authenticating to the cluster when creating the secret. I tried 2 different Terraform Kubernetes provider configurations. Here is the main configuration:

variable "client_id" {}
variable "client_secret" {}

resource "azurerm_resource_group" "rg-example" {
  name     = "rg-example"
  location = "East US"
}

resource "azurerm_kubernetes_cluster" "k8s-example" {
  name                = "k8s-example"
  location            = azurerm_resource_group.rg-example.location
  resource_group_name = azurerm_resource_group.rg-example.name
  dns_prefix          = "k8s-example"

  default_node_pool {
    name            = "default"
    node_count      = 1
    vm_size         = "Standard_B2s"
  }

  service_principal {
    client_id     = var.client_id
    client_secret = var.client_secret
  }

  role_based_access_control {
    enabled = true
  }
}

resource "kubernetes_secret" "secret_example" {
  metadata {
    name = "mysecret"
  }
  data = {
    "something" = "super secret"
  }
  depends_on = [
    azurerm_kubernetes_cluster.k8s-example
  ]
}

provider "azurerm" {
  version = "=2.29.0"
  features {}
}

output "host" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.host
}
output "cluster_username" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.username
}
output "cluster_password" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.password
}
output "client_key" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.client_key
}
output "client_certificate" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.client_certificate
}
output "cluster_ca_certificate" {
  value = azurerm_kubernetes_cluster.k8s-example.kube_config.0.cluster_ca_certificate
}

Here is the first Kubernetes provider configuration using certificates:

provider "kubernetes" {
  version = "=1.13.2"
  load_config_file = "false"

  host = azurerm_kubernetes_cluster.k8s-example.kube_config.0.host
  
  client_certificate     = azurerm_kubernetes_cluster.k8s-example.kube_config.0.client_certificate
  client_key             = azurerm_kubernetes_cluster.k8s-example.kube_config.0.client_key
  cluster_ca_certificate = azurerm_kubernetes_cluster.k8s-example.kube_config.0.cluster_ca_certificate
}

And the error I'm receiving:

kubernetes_secret.secret_example: Creating...

Error: Failed to configure client: tls: failed to find any PEM data in certificate input

Here is the second Kubernetes provider configuration using HTTP Basic Authorization:

provider "kubernetes" {
  version = "=1.13.2"
  load_config_file = "false"

  host = azurerm_kubernetes_cluster.k8s-example.kube_config.0.host
  
  username = azurerm_kubernetes_cluster.k8s-example.kube_config.0.username
  password = azurerm_kubernetes_cluster.k8s-example.kube_config.0.password
}

And the error I'm receiving:

kubernetes_secret.secret_example: Creating...

Error: Post "https://k8s-example-c4a78c03.hcp.eastus.azmk8s.io:443/api/v1/namespaces/default/secrets": x509: certificate signed by unknown authority

ANALYSIS

I checked the outputs of azurerm_kubernetes_cluster.k8s-example and the data seems valid (username, password, host, etc..) Maybe I need a SSL certificate on my Kubernetes cluster, however I'm am not certain, as I'm new to this. Can someone help me out ?

Answer 1

According to this issue in hashicorp/terraform-provider-kubernetes, you need to use base64decode(). The example that author used:

provider "kubernetes" {
  host = "${google_container_cluster.k8sexample.endpoint}"
  username = "${var.master_username}"
  password = "${var.master_password}"
  client_certificate = "${base64decode(google_container_cluster.k8sexample.master_auth.0.client_certificate)}"
  client_key = "${base64decode(google_container_cluster.k8sexample.master_auth.0.client_key)}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.k8sexample.master_auth.0.cluster_ca_certificate)}"
}

That author said they got the same error as you if they left out the base64decode. You can read more about that function here: https://www.terraform.io/docs/configuration/functions/base64decode.html