Terraform - loop over template

Question

I'm new to terraform and I try to achieve something that could remove the redundoncy of my infrastructure. Basically I have one module that calls child modules like so :

module "jobs" {
  source = "./jobs"

  environment                     = var.environment
  region                          = var.region
  name                            = var.name
  bucket_id                       = module.commons.bucket_id
  bucket_name                     = module.commons.bucket_name
}

Inside the jobs folder, I have multiples files that does the same thing but with different values like so :

resource "aws_s3_object" "test" {
  bucket = var.bucket_id
  key    = "scripts/test_samples.sql"
  source = "../src/main/resources/test_samples.sql"

}

resource "aws_s3_object" "test_folder" {
  bucket = var.bucket_id
  key    = "raw/test_samples/"
}

resource "aws_glue_job" "test_samples" {
  name     = "test_samples"
  role_arn = var.glue_role_arn
  execution_class = "STANDARD"

  execution_property {
    max_concurrent_runs = 1
  }

  command {
    name            = "glueetl"
    script_location = "s3://${var.bucket_name}/${aws_s3_object.simple_jdbc_code.key}"
  }

  connections = [var.databricks_jdbc_connection_name]

  max_retries = 0
  number_of_workers = 2
  worker_type = "G.1X"
  timeout = 10 # minutes
  glue_version = "3.0"
  
etc etc.
}

resource "aws_glue_trigger" "test_schedule" {
  name     = "test_samples"
  schedule = "cron(00 02 ? * SUN *)"
  type     = "SCHEDULED"

  actions {
    job_name = aws_glue_job.test_samples.name
  }
}

Basically, instead of running the exact same file with different values, I would want to create a sort of template of the previous code, applying the same parameters but with different values (a sort of loop that would read some differents set of parameters during the execution).

Is there an easy way to realize that ? I've heard of Terragrunt which can help to factorize some terraform code, can I use it in this case ?

Thanks in advance !

Answer 1

Based on the discussion in the comments, you could create a variable on the module level:

variable "something" {
  type        = list(string)
  description = "List of SQL files to be used."
}

Then, in the module itself you would use a combination of for_each [1] and toset [2] built-in functions:

resource "aws_s3_object" "test" {
  for_each = toset(var.something)
  bucket   = var.bucket_id
  key      = "scripts/${each.key}"
  source   = "../src/main/resources/${each.key}"

}

When calling the module, you would provide the list of file names:

module "jobs" {
  source = "./jobs"

  environment                     = var.environment
  region                          = var.region
  name                            = var.name
  bucket_id                       = module.commons.bucket_id
  bucket_name                     = module.commons.bucket_name
  something                       = ["test_samples1.sql", "test_samples2.sql", "test_samples3.sql"]
}

---

[1] https://developer.hashicorp.com/terraform/language/meta-arguments/for_each

[2] https://developer.hashicorp.com/terraform/language/functions/toset