How to make a HTTP call reaching all instances behind amazon AWS load balancer?

Question

I have a web app which runs behind Amazon AWS Elastic Load Balancer with 3 instances attached. The app has a /refresh endpoint to reload reference data. It need to be run whenever new data is available, which happens several times a week.

What I have been doing is assigning public address to all instances, and do refresh independently (using ec2-url/refresh). I agree with Michael's answer on a different topic, EC2 instances behind ELB shouldn't allow direct public access. Now my problem is how can I make elb-url/refresh call reaching all instances behind the load balancer?

And it would be nice if I can collect HTTP responses from multiple instances. But I don't mind doing the refresh blindly for now.

Answer 1

one of the way I'd solve this problem is by

1. writing the data to an AWS s3 bucket
2. triggering a AWS Lambda function automatically from the s3 write
3. using AWS SDK to to identify the instances attached to the ELB from the Lambda function e.g. using boto3 from python or AWS Java SDK
4. call /refresh on individual instances from Lambda
5. ensuring when a new instance is created (due to autoscaling or deployment), it fetches the data from the s3 bucket during startup
6. ensuring that the private subnets the instances are in allows traffic from the subnets attached to the Lambda
7. ensuring that the security groups attached to the instances allow traffic from the security group attached to the Lambda

the key wins of this solution are

• the process is fully automated from the instant the data is written to s3,
• avoids data inconsistency due to autoscaling/deployment,
• simple to maintain (you don't have to hardcode instance ip addresses anywhere),
• you don't have to expose instances outside the VPC
• highly available (AWS ensures the Lambda is invoked on s3 write, you don't worry about running a script in an instance and ensuring the instance is up and running)

hope this is useful.