I set my load balancer to allow traffic on port 80 to the security group of my EC2 instance, but the load balancer refuses to connect. To do a sanity check, when I allow HTTP traffic from my IP to the load balancer, I'm able to connect seamlessly. I can only access the load balancer from my EC2 instance when I allow all traffic to port 80 on my load balancer. I keep checking back and forth to make sure the security group that the EC2 instance is in is the same as the one that the load balancer should allow, so I know it's not that. What else could be blocking the traffic?
Context: I used a load balancer because I wanted a static address for my EC2 instances to access and I don't want the IP to be dynamic. Basically, behind the load balancer is one coordinator instance that I want to manage jobs for the auto scaling EC2 instances. The load balancer can be internal for the purposes I need it for.
Network traffic between an EC2 instance and a public facing load balancer will go out to the Internet and back, it won't stay within the VPC. Once the request leaves the VPC it loses its association with the source security group, which is why your security group rule isn't working. If the EC2 instance has an Elastic IP address assigned to it, you could try adding that to the load balancer's security group instead.
There is probably a better way to do what you are trying to do without having your EC2 instance access a public load balancer. Perhaps the load balancer could be converted to an internal load balancer? Or maybe there is a better way for this EC2 server to communicate with the resources behind the load balancer directly somehow. You would need to post more information about your system and your desired goal for help with that.