Aws Application Load Balancer not accessible unless All traffic is allowed

Question

I have a setup like this.

1. Application Load Balancer(internet facing) LB1 with entry in Route 53 as loadbalancer1.com

2. LB1 is in security group sg1 which has inbound rule to accept all Https requests.

3. LB1 has target group attached which has an EC2 instance instance1 which belongs to security group sg2

4. sg2 has inbound rule which specifies traffic can only come from sg1

5. I have another EC2 instance instance2 in Security Group sg3. This instance wants to access instance1. It does this by calling the load balancer loadbalancer1.com(on port 443 i.e Https)

This setup works. Now I want to make an improvement. Since loadbalancer1.com will only be accessed from instance2. I want to change inbound rules of security group sg1 to accept traffic only from security group sg3. If I do this, loadbalancer1.com is no more reachable from instance2.

Any idea why?

Answer 1

It's a very good question. TL;DR: it only works for internal traffic that doesn't leave the VPC.

Here is what's going on in more detail:

When the instance2 accesses the internet-facing load balancer, the traffic first leaves your VPC and goes to the public internet. Then the traffic reaches the ELB through some network routings, and the ELB forwards the traffic to the instance1.

When the traffic leaves the AWS VPC and re-enters it, the source SG metadata associated with the traffic is "gone". From the SG of the ELB point of view, it only knows that the traffic is originated from the public IP address of instance2.

According to the Security group rules documentation:

When you specify a security group as the source or destination for a rule, the rule affects all instances that are associated with the security group. Incoming traffic is allowed based on the private IP addresses of the instances that are associated with the source security group (and not the public IP or Elastic IP addresses).