How to handle descrepencies between develop and main branches of Terraform project

Question

There are main and develop branches in TF project.
In develop branch, there could be resources specified with IDs which are dependent on AWS environment, e.g. 2c289348-cc9d-41ed-a4ae-fb015b0ec9b1-6 is AWS MSK cluster id:

resource "aws_iam_policy" "msk-admin-policy" {
  name        = "msk-admin-policy"
  description = "Policy for MSK"

  policy = jsonencode({
    Version   = "2012-10-17",
    "Statement": [
      {

        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
          <...actions>
        ],
        "Resource": [
          "arn:aws:kafka:${data.aws_region.current.id}:${data.aws_caller_identity.current.account_id}:cluster/integrations/2c289348-cc9d-41ed-a4ae-fb015b0ec9b1-6",
        ]
      }
    ]
  })
} 

In main branch, there is different AWS MSK id which stands for id of AWS MSK cluster in production AWS environment.

It leads us to manage develop and main branches separately, meaning that we are not able to do a marge request from develop to main. Instead, we are creating a feature branch from develop, then create a merge request from a feature branch to develop. Then we create feature branch from main, create a merge request from a feature branch to main

As you can see, we have applied kind of unification between branches for region and account id by ${data.aws_region.current.id}:${data.aws_caller_identity.current.account_id}.

Is there an approach to handle such kind of discrepencies between main and develop branches in TF?

Answer 1

You shouldn't be hard-coding those values at all, you should be referencing the Terraform resource, or looking up the datasource. If you absolutely have to hard-code the value, then I recommend hard-coding it in a Terraform map variable, like:

variable "msk_id" {
  description = "the MSK Cluster ID"

  type = map(string)
  default = {
    dev: "2c289348-cc9d-41ed-a4ae-fb015b0ec9b1-6"
    production : "some-other-id"
  }
}

Then you could pass the environment name as another variable, or ideally use Terraform Workspaces to manage your different environments.

Then your IAM policy would look like this (with the environment name in a var):

"Resource": [
  "arn:aws:kafka:${data.aws_region.current.id}:${data.aws_caller_identity.current.account_id}:cluster/integrations/${var.msk_id[var.environment]}",
]

or (with Terraform workspaces):

"Resource": [
  "arn:aws:kafka:${data.aws_region.current.id}:${data.aws_caller_identity.current.account_id}:cluster/integrations/${var.msk_id[terraform.workspace]}",
]