Terraform AWS -Using Data from remote backend issue

Question

I am currently building a 3 tier wordpress structure for a project that includes VPC, ASG-ALB, EFS and RDS clusters. .TF files are in seperate directories so i am using makefile to apply them in order. Backend is stored in s3 bucket/dynamodb.

I want to be able to get certain info from RDS, EFS tf files that are located in seperate directories under same project so i can use them in userdata template under ASG.tf file.

Please see below:

My user_data template requires below vars:

data "template_file" "user_data" {
  template = file("user_data.sh")
  vars = {
    db_username      = var.database_user
    **db_user_password** = data.random_string.rds_password.result ## retreives from random_string resource in rds.tf in seperate folder.
    db_name          = var.database_name
    **db_RDS**           = data.aws_rds_cluster.wordpress_db_cluster.endpoint ## retreives from rds_cluster.wordpress resource in rds.tf in seperate folder.
    **efs_dns_name**     = data.aws_efs_file_system.efs.dns_name ## retreives from aws_efs_file_system resource in efs.tf in seperate folder.
  }
}

EFS and RDS are already been built and ready to use.

i get below errors when trying to retreive db_user_password(created using random string under RDS directory), db_RDS(need cluster endpoint) and efs_dns_name(need endpoint from efs).

│ Error: Reference to undeclared resource
│ 
│   on main.tf line 82, in data "template_file" "user_data":
│   82:     db_user_password = data.random_string.rds_password.result ## retreive from random_string resource in rds.tf in RDS folder.
│ 
│ A data resource "random_string" "rds_password" has not been declared in the root module.
╵
╷
│ Error: Reference to undeclared resource
│ 
│   on main.tf line 84, in data "template_file" "user_data":
│   84:     db_RDS           = data.aws_rds_cluster.wordpress_db_cluster.endpoint ## retreive from rds_cluster.wordpress resource in rds.tf in RDS folder.
│ 
│ A data resource "aws_rds_cluster" "wordpress_db_cluster" has not been declared in the root module.
╵
╷
│ Error: Reference to undeclared resource
│ 
│   on main.tf line 85, in data "template_file" "user_data":
│   85:     efs_dns_name     = data.aws_efs_file_system.efs.dns_name ## retreive from aws_efs_file_system resource in efs.tf in EFS folder.
│ 
│ A data resource "aws_efs_file_system" "efs" has not been declared in the root module.

My question is how can i declare above resources that are from another directory to use inside my template. Or is there another way around to use them. I have tried adding

data.terraform_remote_state.backend

but still no luck.

Answer 1

There are three different methods to achieve solution to your problem.

Solution nr 1 - Terraform modules

The components of the system you have created in separate directiories became separate Terraform projects. The intended way of stucturing such dependency is to use Terraform modules.

In this scenario, the entire project is called root module, and all components are in subdirectiories (you can think of them as subprojects). Any argument to the module is treated as variable in the subproject (module), and any data needed back in the root module (for example to be used in another module) should be outputted from this subproject.

Solution nr 2 - Remote state data objects

If you choose to leave the components as totally separate projects, you can still read their remote state (and, in particular, their outputs) using Remote State Data. In this approach, you should remember to output necessary attributes in the source module, and use those with remote state data as shown in tutorial I have linked to.

Solution nr 3 - Terragrunt

The approach you have come up with is particularly similar to the approach that is proposed by Gruntwork's Terragrunt. If you see the quickstart of terragrunt, you may find that this tool is about splitting terraform project into subprojects just like you do, and running them all using dependency statement, in proper order and more- probably the same way you created your Makefile for.

I favor the solution nr 1, but I would look and think about all of them to choose the one that suits you most. Good luck!