How should a kubernetes service account and token be configured in Terraform to avoid a race condition?

Question

I've inherited some kubernetes/minikube infrastructure and I'm experiencing an issue where there seems to be a race condition with a service account token being created. This is what we have for the terraform code:

resource "kubernetes_service_account_v1" "foo-service-account" {
  metadata {
    name = "foo-service-account"
    namespace = "foo-${var.cluster_env}"
  }
  automount_service_account_token = true
}
    
resource "kubernetes_secret_v1" "foo-service-account-token" {
  metadata {
    name = "foo-service-account-token"
    namespace = "foo-${var.cluster_env}"
    annotations = {
      "kubernetes.io/service-account.name" = "foo-service-account"
   }
 }
  type = "kubernetes.io/service-account-token"
  wait_for_service_account_token = true
}

Roughly 50% of the time, the Makefile commands we run to bring up the apps locally result in an error:

╷
│ Error: secrets "foo-service-account-token" not found
│
│ with kubernetes_secret_v1.foo-service-account-token,
│ on foo.tf line 36, in resource "kubernetes_secret_v1" "foo-service-account-token":
│ 36: resource "kubernetes_secret_v1" "foo-service-account-token" { │
╵

We've tried changing the values of automount_service_account_token and wait_for_service_account_token as well as removing one, the other, or both, and nothing has consistently fixed the issue. Is there something else we should be doing to alleviate the problem?

Answer 1

How about using depends_on argument?

With it your resources will looks like this:

resource "kubernetes_service_account_v1" "foo-service-account" {
  metadata {
    name = "foo-service-account"
    namespace = "foo-${var.cluster_env}"
  }
  automount_service_account_token = true
}

resource "kubernetes_secret_v1" "foo-service-account-token" {
  depends_on = [
    kubernetes_service_account_v1.foo-service-account
  ]

  metadata {
    name = "foo-service-account-token"
    namespace = "foo-${var.cluster_env}"
    annotations = {
      "kubernetes.io/service-account.name" = "foo-service-account"
    }
  }
  type = "kubernetes.io/service-account-token"
  wait_for_service_account_token = true
}

That should prevent race condition since secret will always wait for service account.