Circular Dependency error with ECS accessing RDS

Question

I've created ECS fargate connected to RDS postgresql. Load balancer is public facing, and RDS is ISOLATED. I want RDS database accessed through ECS container/load balancer and EC2. Once the RDS instance is created, I added ingress rule to allow security group for ecs.

props.dbInstance.connections.securityGroups[0].addIngressRule(
  ecsSecurityGroup,
  ec2.Port.tcp(5432),
  'Allow ECS access to RDS Postgres'
);

However, I get a Circular Dependency Error.

❌ CdkStack failed: Error [ValidationError]: Circular dependency between resources: [RDSNestedStackRDSNestedStackResource4BEA771C, ECSNestedStackECSNestedStackResourceFF50F200]

There are two stacks: RDS and ECS.

RDSStack,

export class RDSStack extends NestedStack {
  public readonly dbInstance: rds.DatabaseInstance;

  constructor(scope: Construct, id: string, props: RDSStackProps) {
    super(scope, id);

    // RDS security group
    const rdsSecurityGroup = new ec2.SecurityGroup(this, `RDSSecurityGroup`, {
      vpc: props.vpc,
      description: 'Security group for RDS instance',
    });

    // New private subnet group for RDS instance
    const rdsSubnetGroup = new rds.SubnetGroup(this, `RDSInstanceSubnetGroup`, {
      vpc: props.vpc,
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
      description: 'Subnet group for RDS instance',
    });

    // RDS instance in the private subnet with no public access
    this.dbInstance = new rds.DatabaseInstance(this, `RDSInstance`, {
      publiclyAccessible: false,
      subnetGroup: rdsSubnetGroup,
      vpc: props.vpc,
      port: 5432,
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
      securityGroups: [rdsSecurityGroup],
      // more
    });

    // EC2 security group 
    const ec2SecurityGroup = new ec2.SecurityGroup(this, `EC2SecurityGroup`, {
      vpc: props.vpc,
      description: 'Security group for EC2 instance',
    });

    // Allow SSM traffic (HTTPS)
    ec2SecurityGroup.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(443), 'Allow HTTPS traffic for SSM');

    // Allow all outbound connections for EC2 instance
    ec2SecurityGroup.addEgressRule(ec2.Peer.anyIpv4(), ec2.Port.allTraffic());

    // ingress rules to RDS security group to allow traffic from the EC2
    rdsSecurityGroup.addIngressRule(
      ec2.Peer.securityGroupId(ec2SecurityGroup.securityGroupId),
      ec2.Port.tcp(5432),
      'Allow DB access from EC2 instance'
    );

and ECSStack

export class ECSStack extends NestedStack {
  constructor(scope: Construct, id: string, props: ECSStackProps) {
    super(scope, id, props);

    const ecsSecurityGroup = new ec2.SecurityGroup(this, `ECSSecurityGroup`, {
      vpc: props.vpc,
      description: 'Security group for ECS',
    });

    const cluster = new ecs.Cluster(this, `ECSCluster`, { vpc });
    const taskDefinition = //...
    const container = //...

    const albfs = new ecs_patterns.ApplicationLoadBalancedFargateService(this, `FargateService`,
      {
        publicLoadBalancer: true,
        securityGroups: [ecsSecurityGroup],
        assignPublicIp: true,
        // more
      }
    );

    const lbSecurityGroup = albfs.loadBalancer.connections.securityGroups[0];
    lbSecurityGroup.addIngressRule(
      ec2.Peer.anyIpv4(),
      ec2.Port.tcp(80), // 443 for https
      'Allow HTTP from anywhere'
    );

    ----> Problem line of codes 
    // Grant permissions to ECS task to access RDS
    props.dbInstance.connections.securityGroups[0].addIngressRule(
      ecsSecurityGroup,
      ec2.Port.tcp(5432),
      'Allow ECS access to RDS Postgres'
    );
}

If I comment out the problem lines, I can deploy but the ECS containers do not seem to be accessing database. Why am I getting a circular error? I want ECS containers to access RDS.

Answer 1

First of all - you don't need to create or deal with Security Groups yourself - they're considered an implementation detail. From the docs:

Direct manipulation of the Security Group through addIngressRule and addEgressRule is possible, but mutation through the .connections object is recommended. If you peer two constructs with security groups this way, appropriate rules will be created in both.

Instead, you would only use the Connections class, available via the connections property - just like you're already doing, but without dealing with SGs. The following code is roughly equivalent to your code:

props.dbInstance.connections.allowDefaultPortFrom(albfs.service);

Now, the reason for the circular dependency and the solution for it are called out in the aws_ec2 module Readme:

If you are attempting to add a connection from a peer in one stack to a peer in a different stack, sometimes it is necessary to ensure that you are making the connection in a specific stack in order to avoid a cyclic reference. If there are no other dependencies between stacks then it will not matter in which stack you make the connection, but if there are existing dependencies (i.e. stack1 already depends on stack2), then it is important to make the connection in the dependent stack (i.e. stack1).

Whenever you make a connections function call, the ingress and egress security group rules will be added to the stack that the calling object exists in. So if you are doing something like peer1.connections.allowFrom(peer2), then the security group rules (both ingress and egress) will be created in peer1's Stack.

This means that your RDS stack depends on the ECS stack (since the rules are being created in the RDS stack), and the ECS stack depends on the RDS stack presumably because you're passing the instance connection info to the container (omitted from the code in the question).

To solve this, you need to create the rules in the ECS stack. Thankfully, the Connections class provides a lot of flexibility here:

albfs.service.connections.allowToDefaultPort(props.dbInstance);

This would create the rules in the ECS stack, preventing the RDS stack from depending on it.