loop across modules in terraform

I need to build about 30 pub sub topics in GCP, creating each module for a pub sub topic is a tedious process, is there any better way for handling it ?

module "a" {
  source       = ""
  project_id   = var.project_id
  topic        = var.a["topic_name"]
  topic_labels = var.a["topic_labels"]
  pull_subscriptions = [
    {
      name                    = var.a["pull_subscription_name"]
      ack_deadline_seconds    = var.a["ack_deadline_seconds"]
      max_delivery_attempts   = var.a["max_delivery_attempts"]
      maximum_backoff         = var.maximum_backoff
      minimum_backoff         = var.minimum_backoff
      expiration_policy       = var.expiration_policy
      enable_message_ordering = true
    }
  ]
}

module "b" {
  source       = ""
  project_id   = var.project_id
  topic        = var.b["topic_name"]
  topic_labels = var.b["topic_labels"]
  pull_subscriptions = [
    {
      name                    = var.b["pull_subscription_name"]
      ack_deadline_seconds    = var.b["ack_deadline_seconds"]
      max_delivery_attempts   = var.b["max_delivery_attempts"]
      maximum_backoff         = var.maximum_backoff
      minimum_backoff         = var.minimum_backoff
      expiration_policy       = var.expiration_policy
      enable_message_ordering = true
    }
  ]
}

In tfvars passing the values to the above modules like below:

a = {
  topic_name             = "abc"
  topic_labels           = { env : "prod", purpose : "a" }
  pull_subscription_name = "abc-sub"
  ack_deadline_seconds   = 600
  max_delivery_attempts  = 3
}

b = {
  topic_name             = "bcd"
  topic_labels           = { env : "prod", purpose : "b" }
  pull_subscription_name = "bcd-sub"
  ack_deadline_seconds   = 600
  max_delivery_attempts  = 3
}

Can we somehow combine the variables in tfvars and pass in to a single module ?

I also wanna know the best practise to maintain the above terraform script to keep them individually or utilise one module to created 50 topics ?

Thank You !

Solution

With two separate variables your options are a bit limited, because Terraform can't see those two separate variables as being connected in a systematic way. (Each variable, as with other objects in Terraform, is entirely separate from a dependency-resolving standpoint.)

However, if you can restructure this to be a single variable of a map type then you can use resource for_each to systematically declare an instance for each element of the map:

variable "topics" {
  type = map(object({
    labels                 = map(string)
    pull_subscription_name = string
    ack_deadline_seconds   = number
    max_delivery_attempts  = number
  }))
}

module "topic" {
  source   = "..."
  for_each = var.topics

  project_id   = var.project_id
  topic        = each.key
  topic_labels = each.value.labels
  pull_subscriptions = [
    {
      name                    = each.value.pull_scription_name
      ack_deadline_seconds    = each.value.ack_deadline_seconds
      max_delivery_attempts   = each.value.max_delivery_attempts
      maximum_backoff         = var.maximum_backoff
      minimum_backoff         = var.minimum_backoff
      expiration_policy       = var.expiration_policy
      enable_message_ordering = true
    }
  ]
}

For this example I assumed that your topic_name attribute would be a suitable unique key for instances so I removed it from the declared object type with the intent of putting it in the map key instead. In other words, the value for this topics variable should look like this:

topics = {
  abc = {
    labels                 = { env = "prod", purpose = "a" }
    pull_subscription_name = "abc-sub"
    ack_deadline_seconds   = 600
    max_delivery_attempts  = 3
  }
  bcd = {
    labels                 = { env = "prod", purpose = "b" }
    pull_subscription_name = "bcd-sub"
    ack_deadline_seconds   = 600
    max_delivery_attempts  = 3
  }
}

From this, Terraform will understand that you intend to declare module instances with the following addresses:

module.topic["abc"]
module.topic["bcd"]

Because the topic name is part of the address, Terraform can recognize the difference between editing an existing topic object (without changing its name) and adding/removing topics from the map, translating that to the corresponding plan with the resource instances declared inside the module.

If it's important that you be able to change the topic names that the remote system knows without Terraform understanding that as separate delete and create operations, you could restore name as an attribute of that object type and then set topic = each.value.name instead, and then the map keys will only be for tracking in Terraform, and not visible in the remote system at all.