Github Actions and Creating an S3 bucket to use in the RDS creation

Question

I am using Terraform in a GitHub Action to launch an RDS instance to AWS. When creating the instance, I get the following message:

Error: Failed to get existing workspaces: S3 bucket "***" does not exist.

This happens because I am using the bucket to hold state info and am creating it with the Terraform backend keywork.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 5.40.0"
    }
  }

  required_version = ">= 1.7.4"

  backend "s3" {
    bucket = "removed_for_question"
    key = "removed_for_question"
    region = "removed_for_question"
  }
}

After some research, I found out this is called the chicken-egg problem of Terraform? I tried solving it a multitude of ways, but unfortunately I am fairly new at most of the technologies I've listed and can't seem to get a solution that will solve my specific problem.

How does one go about creating the S3 bucket in a Github Action before the RDS Terraform creation runs that will need this bucket. I know you can use the AWS CLI to create it as well, but am not too sure of the best practice here.

How does the industry solve this issue, I'm struggling to believe everyone creates the bucket manually.

Answer 1

Your comment here sounds like you think declaring an S3 backend will actually lead to Terraform creating the S3 bucket:

I am using the bucket to hold state info and am creating it with the Terraform backend keywork.

That is not the case. Terraform will not create the S3 bucket you are using as a backend. You have to create the S3 bucket yourself, before using this Terraform code. There is no way around this. If you wanted this S3 bucket to be managed by Terraform, then you would have to separate it out into a separate Terraform template that you run before this Terraform code, but then you would have the same problem with state storage in the new Terraform code.

If you want to automate this whole process in your GitHub Actions, then you could use the AWS CLI Tool to create the S3 bucket before running the Terraform code. You would have to put some sort of logic in your GitHub Actions to check if the bucket exists, and only run the CLI commands to create the bucket once, as that will fail if the bucket already exists.