AWS CloudFormation: Target Group for Application Load Balancer is not working for multiple EC2 instances

Question

I am deploying my infrastructure to AWS using CloudFormation template. My infrastructure has an application load balancer that is pointing to a target group. The target group will have multiple EC2 instances in it.

The following is my target group and the EC2 instances in the template.

Resources:
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 0, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 1, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 1
        - Fn::GetAZs: !Ref AWS::Region
  PublicSubnet3:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 2, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 2
        - Fn::GetAZs: !Ref AWS::Region
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref Vpc
      InternetGatewayId: !Ref InternetGateway
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref Vpc
  Route:
    Type: AWS::EC2::Route
    DependsOn: InternetGateway
    Properties:
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref RouteTable
  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Application Load Balancer Security Group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '443'
          ToPort: '443'
          CidrIp: 0.0.0.0/0
      VpcId: !Ref Vpc
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: ApplicationLoadBalancer
      Subnets:
        - !Ref PublicSubnet1
        - !Ref PublicSubnet2
        - !Ref PublicSubnet3
      SecurityGroups:
        - !Ref LoadBalancerSecurityGroup
  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref ApplicationTargetGroup
  ApplicationTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 15
      HealthyThresholdCount: 5
      UnhealthyThresholdCount: 3
      Matcher:
        HttpCode: '200'
      Name: ApplicationTargetGroup
      VpcId: !Ref Vpc
      Port: 80
      Protocol: HTTP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: '20'
      Targets:
        - Id: !Ref WebServerInstance1
          Port: 80
        - Id: !Ref WebServerInstance2
            Port: 80
  WebServerInstance1:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SubnetId: !Ref PublicSubnet1
      SecurityGroupIds:
        - !Ref WebServerSecurityGroup
      ImageId:
        Fn::FindInMap:
          - AWSRegionArch2AMI
          - Ref: AWS::Region
          - Fn::FindInMap:
              - AWSInstanceType2Arch
              - Ref: InstanceType
              - Arch
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -xe
          cd /tmp
          yum update -y
          yum install -y httpd24
          echo "Welcome from the instance 1" > /var/www/html/index.html
          sudo -u root service httpd start
  WebServerInstance2:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SubnetId: !Ref PublicSubnet1
      SecurityGroupIds:
        - !Ref WebServerSecurityGroup
      ImageId:
        Fn::FindInMap:
          - AWSRegionArch2AMI
          - Ref: AWS::Region
          - Fn::FindInMap:
              - AWSInstanceType2Arch
              - Ref: InstanceType
              - Arch
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -xe
          cd /tmp
          yum update -y
          yum install -y httpd24
          echo "Welcome from the instance 2" > /var/www/html/index.html
          sudo -u root service httpd start
  WebServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable HTTP access via port 80
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp:
            Ref: SSHLocation
      VpcId: !Ref Vpc

As you can see my target group has two targets. But when I deployed and open the load balancer DNS in the browser, it just keeps loading, loading and loading. But if I assign only one EC2 instance to the target group, it is working as expected. It is just not working for multiple instances. What is wrong with my template and how can I fix it?

Answer 1

I can confirm that the route cause is that only one of the public subnets actually has a route table association for a route table with an internet gateway. Add a subnet association for each subnet and it should go away.

This is because traffic could not be routed for the subnets that did not have the association so the connection hung.

Below is the working template

Resources:
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 0, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 1, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 1
        - Fn::GetAZs: !Ref AWS::Region
  PublicSubnet3:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref Vpc
      CidrBlock: !Select [ 2, !Cidr [ !Ref VpcCidr, 12, 8 ] ]
      MapPublicIpOnLaunch: True
      AvailabilityZone: !Select
        - 2
        - Fn::GetAZs: !Ref AWS::Region
  InternetGateway:
    Type: AWS::EC2::InternetGateway
  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref Vpc
      InternetGatewayId: !Ref InternetGateway
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref Vpc
  Route:
    Type: AWS::EC2::Route
    DependsOn: InternetGateway
    Properties:
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
  SubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref RouteTable
  SubnetRouteTableAssociationB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref RouteTable
  SubnetRouteTableAssociationC:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet3
      RouteTableId: !Ref RouteTable
  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Application Load Balancer Security Group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '443'
          ToPort: '443'
          CidrIp: 0.0.0.0/0
      VpcId: !Ref Vpc
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: ApplicationLoadBalancer
      Subnets:
        - !Ref PublicSubnet1
        - !Ref PublicSubnet2
        - !Ref PublicSubnet3
      SecurityGroups:
        - !Ref LoadBalancerSecurityGroup
  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref ApplicationTargetGroup
  ApplicationTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 15
      HealthyThresholdCount: 5
      UnhealthyThresholdCount: 3
      Matcher:
        HttpCode: '200'
      Name: ApplicationTargetGroup
      VpcId: !Ref Vpc
      Port: 80
      Protocol: HTTP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: '20'
      Targets:
        - Id: !Ref WebServerInstance1
          Port: 80
        - Id: !Ref WebServerInstance2
          Port: 80
  WebServerInstance1:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SubnetId: !Ref PublicSubnet1
      SecurityGroupIds:
        - !Ref WebServerSecurityGroup
      ImageId:
        Fn::FindInMap:
          - AWSRegionArch2AMI
          - Ref: AWS::Region
          - Fn::FindInMap:
              - AWSInstanceType2Arch
              - Ref: InstanceType
              - Arch
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -xe
          cd /tmp
          yum update -y
          yum install -y httpd24
          echo "Welcome from the instance 1" > /var/www/html/index.html
          sudo -u root service httpd start
  WebServerInstance2:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyName
      SubnetId: !Ref PublicSubnet1
      SecurityGroupIds:
        - !Ref WebServerSecurityGroup
      ImageId:
        Fn::FindInMap:
          - AWSRegionArch2AMI
          - Ref: AWS::Region
          - Fn::FindInMap:
              - AWSInstanceType2Arch
              - Ref: InstanceType
              - Arch
      AvailabilityZone: !Select
        - 0
        - Fn::GetAZs: !Ref AWS::Region
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -xe
          cd /tmp
          yum update -y
          yum install -y httpd24
          echo "Welcome from the instance 2" > /var/www/html/index.html
          sudo -u root service httpd start
  WebServerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable HTTP access via port 80
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp:
            Ref: SSHLocation
      VpcId: !Ref Vpc
```