Terraform http datasource to generate dynamic variables at terraform plan and apply

Question

I have a eventbridge.tf where I am using the cron_schedule_info variable data from my env.tfvars, currently which is kind of hardcoded like this for different environments.

variables.tf

variable "cron_schedule_info" {
  type = map(list(object({
    property_id         = string
    schedule_expression = string
  })))
}

env.tfvars

cron_schedule_info = {
  AAA = [
    {
      property_id         = "property1",
      schedule_expression = "cron(0 * * * ? *)"
    },
    {
      property_id         = "property2",
      schedule_expression = "cron(5 * * * ? *)"
    },
    ...........
    ...........
    {
      property_id         = "property100",
      schedule_expression = "cron(55 * * * ? *)"
    }
  ]
  BBB = [
    {
      property_id         = "property1",
      schedule_expression = "cron(0 * * * ? *)"
    }
  ]
}

eventbridge.tf

locals {
  rule_list = flatten([
    for i, p in var.cron_schedule_info : [
      for index, data in p :
      merge(data, {
        "i" = i
      })
    ]
  ])

  rule = {
    for index, rule in local.rule_list :
    "prule_${rule.i}_${index}" => {
      description         = "Trigger cron"
      schedule_expression = rule.schedule_expression
    }
  }
}

My current system works fine as expected where I have defined a terraform variable with map type and I add manually EventBridge rule in infrastructure definitions by hardcodeding the value everytime I introduce a new item like above cron_schedule_info. Now I am planning to store this information in some sort of datastore, and have such EventBridge rules be generated dynamically accordingly at Terraform 'apply' or 'plan' time.

So my initial thought is I wll store the above data as a table where I can add, update, delete. My table structure will be like this-

 id | integration_type | property_id |   schedule_expression   | cron_type | active
----+-------------------+-------------+-------------------------+-----------+-------
  1 | AAA               | property1   | cron(0 * * * ? *)       | type1    | false
  2 | AAA               | property2   | cron(5 * * * ? *)       | type2    | true
  3 | AAA               | property20  | cron(55 * * * ? *)      | type3    | true
  4 | BBB               | property1   | cron(0 * * * ? *)       | type1    | true

This will give me flexiblity to change my cron information any time regardless to change on my terraform env.tfvars, then at the time of terraform apply or plan I will get my cron_schedule_info as json response from an api endpoint that I will build further.

assumed json response

{
  "cron_schedule_info": {
    "AAA": [
      {
        "property_id": "property1",
        "schedule_expression": "cron(0 * * * ? *)"
      },
      {
        "property_id": "property2",
        "schedule_expression": "cron(5 * * * ? *)"
      },
      {
        "property_id": "property100",
        "schedule_expression": "cron(55 * * * ? *)"
      }
    ],
    "BBB": [
      {
        "property_id": "property1",
        "schedule_expression": "cron(0 * * * ? *)"
      }
    ]
  }
}

So I am planing to use terraform http data source for this problem, ref: https://github.com/hashicorp/terraform-provider-http/blob/main/docs/data-sources/http.md in that way I can fetch the api response from my endpoint for the terraform 'plan' or 'apply'.

• Q1. Is my approach is correct or any other better solution exists for this types of problem?
• Q2. If I use api endpoint for the remote data source in terraform http datasource, how can I further convert or use that response in my existing code e.g how to process or how to modify my existing code.

Answer 1

Q1.Response:

Yes you are approach is correct.

Q2.Response:

"If the API response has the same structure as your 'cron_schedule_info' variable." then you can replace your local map interaction var.cron_schedule_info by data.http.cron_schedule_info as the next example:

---------- data source

data "http" "cron_schedule_info" {
  url = "https://cron_schedule_info-api/v1/"

  # Optional request headers
  request_headers = {
    Accept = "application/json"
  }
}

response example as you expect

{
  "cron_schedule_info": {
    "AAA": [
      {
        "property_id": "property1",
        "schedule_expression": "cron(0 * * * ? *)"
      },
      {
        "property_id": "property2",
        "schedule_expression": "cron(5 * * * ? *)"
      },
      {
        "property_id": "property100",
        "schedule_expression": "cron(55 * * * ? *)"
      }
    ],
    "BBB": [
      {
        "property_id": "property1",
        "schedule_expression": "cron(0 * * * ? *)"
      }
    ]
  }
}

-------------------- locals

locals {
  rule_list = flatten([
    for i, p in data.http.cron_schedule_info : [
      for index, data in p :
      merge(data, {
        "i" = i
      })
    ]
  ])

  rule = {
    for index, rule in local.rule_list :
    "prule_${rule.i}_${index}" => {
      description         = "Trigger cron"
      schedule_expression = rule.schedule_expression
    }
  }
}