How do I properly set up a cross-region transit gateway route with terraform?

Question

I'm trying to set up a transit gateway (TGW) that spans two regions via terraform. I set up the two TGWs on either region, create a TGW attachment in either region, a route table with routes pointing to the opposite region with a defined subnet, but at creation time, there is an inherent default route table that points to the local subnet that messes with the routing. Here is a snippet of my terraform:

   resource "aws_ec2_transit_gateway" "main" {
    description = "main transit gateway"
    provider = aws

    tags = {
      Name = "main draas transit gateway"
    }
  }

  resource "aws_ec2_transit_gateway_vpc_attachment" "main" {
    subnet_ids = [aws_subnet.private.id]
    transit_gateway_id = aws_ec2_transit_gateway.main.id
    vpc_id = aws_vpc.main.id
    provider = aws

    tags = {
      Name = "main draas tgw attachment"
    }
  }

  resource "aws_ec2_transit_gateway_route_table" "main" {
    transit_gateway_id = aws_ec2_transit_gateway.main.id
    provider = aws

    tags = {
      Name = "main draas tgw route table"
    }
  }

  resource "aws_ec2_transit_gateway_route" "main" {
    destination_cidr_block = var.staging_private_subnet_cidr
    transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.main.id
    transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.main.id
    provider = aws
  }

  resource "aws_ec2_transit_gateway_peering_attachment" "main" {
    provider = aws
    peer_transit_gateway_id = aws_ec2_transit_gateway.dr.id
    transit_gateway_id = aws_ec2_transit_gateway.main.id
    peer_account_id = var.account_id
    peer_region = var.dr_region

    tags = {
      Name = "main draas tgw peering"
    }
  }

resource "aws_ec2_transit_gateway" "dr" {
  provider = aws.dr
  description = "dr transit gateway"

  tags = {
    Name = "dr transit draas gateway"
  }
}

resource "aws_ec2_transit_gateway_vpc_attachment" "dr" {
  provider = aws.dr
  subnet_ids = [aws_subnet.staging_private.id]
  transit_gateway_id = aws_ec2_transit_gateway.dr.id
  vpc_id = aws_vpc.staging.id

  tags = {
    Name = "dr tgw draas attachment"
  }
}

resource "aws_ec2_transit_gateway_route_table" "dr" {
  provider = aws.dr
  transit_gateway_id = aws_ec2_transit_gateway.dr.id

  tags = {
    Name = "dr tgw draas route table"
  }
}

resource "aws_ec2_transit_gateway_route" "dr" {
  provider = aws.dr
  destination_cidr_block = var.main_private_subnet_cidr
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.dr.id
  transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.dr.id
}

When I do a terraform apply, I get two TGW route tables in each region, the default, which points back to the CIDR that is associated with the TGW in the same region, and the one I create here in my terraform code, which should be routing traffic to the opposing region.

When I do a reachability analysis, an ec2 instance in us-east-1 cannot ping one in us-west-2 and the TGW route table is the problem. It seems as though the default TGW route table is being used while the one I create in terraform is ignored. I realize the way I accomplish peering in my .tf code isn't correct, but even after I accept peering in the console, the routing is incorrect.

I know there are some limitations around terraform, but does anyone have any clever ways around this besides going into the AWS console and manually changing things?

Answer 1

It looks like if you turn the gateway route tables to point to data objects, it works. Confirmed with VPC Reachability Analyzer. I'll post updated code soon.

EDIT: Updated with working Terraform:

  resource "aws_ec2_transit_gateway" "main" {
    description = "main transit gateway"
    provider = aws

    tags = {
      Name = "main draas transit gateway"
    }
  }

  resource "aws_ec2_transit_gateway_vpc_attachment" "main" {
    subnet_ids = [aws_subnet.private.id]
    transit_gateway_id = aws_ec2_transit_gateway.main.id
    vpc_id = aws_vpc.main.id
    provider = aws

    tags = {
      Name = "main draas tgw attachment"
    }
  }

  data "aws_ec2_transit_gateway_route_table" "main" {
    provider = aws

    filter {
      name   = "default-association-route-table"
      values = ["true"]
    }

    filter {
      name   = "transit-gateway-id"
      values = [aws_ec2_transit_gateway.main.id]
    }

    tags = {
      Name = "main tgw draas route table"
    }
  }

  resource "aws_ec2_transit_gateway_route" "main" {
    destination_cidr_block = var.staging_private_subnet_cidr
    transit_gateway_route_table_id = data.aws_ec2_transit_gateway_route_table.main.id
    transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.main.id
    provider = aws
  }

  resource "aws_ec2_transit_gateway_peering_attachment" "main" {
    provider = aws
    peer_transit_gateway_id = aws_ec2_transit_gateway.dr.id
    transit_gateway_id = aws_ec2_transit_gateway.main.id
    peer_account_id = var.account_id
    peer_region = var.dr_region

    tags = {
      Name = "main draas tgw peering"
    }
  }

## DR

resource "aws_ec2_transit_gateway" "dr" {
  provider = aws.dr
  description = "dr transit gateway"

  tags = {
    Name = "dr transit draas gateway"
  }
}

resource "aws_ec2_transit_gateway_vpc_attachment" "dr" {
  provider = aws.dr
  subnet_ids = [aws_subnet.staging_private.id]
  transit_gateway_id = aws_ec2_transit_gateway.dr.id
  vpc_id = aws_vpc.staging.id

  tags = {
    Name = "dr tgw draas attachment"
  }
}

data "aws_ec2_transit_gateway_route_table" "dr" {
  provider = aws.dr

  filter {
    name   = "default-association-route-table"
    values = ["true"]
  }

  filter {
    name   = "transit-gateway-id"
    values = [aws_ec2_transit_gateway.dr.id]
  }

  tags = {
    Name = "dr tgw draas route table"
  }
}

resource "aws_ec2_transit_gateway_route" "dr" {
  provider = aws.dr
  destination_cidr_block = var.main_private_subnet_cidr
  transit_gateway_route_table_id = data.aws_ec2_transit_gateway_route_table.dr.id
  transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.dr.id
}

data "aws_ec2_transit_gateway_peering_attachment" "dr" {
  provider = aws.dr
  depends_on = [ aws_ec2_transit_gateway_peering_attachment.main ]

  filter {
    name = "state"
    values = [ "pendingAcceptance" ]
  }

  # Only the second accepter/peer transit gateway is called from the peering attachment.
  filter {
    name = "transit-gateway-id"
    values = [ aws_ec2_transit_gateway_peering_attachment.main.peer_transit_gateway_id ]
  }
}

resource "aws_ec2_transit_gateway_peering_attachment_accepter" "dr" {
  provider = aws.dr
  transit_gateway_attachment_id = data.aws_ec2_transit_gateway_peering_attachment.dr.id
}