I am facing an interesting problem. We run an HTTPS based application that is state based. State is maintained based on session cookie. The application is designed in such a way that if the session gets terminated abruptly, the application is reverted to home screen any un-saved data is lost. So its very important for us to maintain session.
Some point in the past it was decided to use AWS services for this purpose and the current architecture has an ELB that balances load to an auto-scaling group. The first architecture that was used had HTTP based sticky session enabled. It is found out during testing that when scaling down the existing sessions are getting closed immediately and gets re-routed to available instances. This happens even after enabling draining (time out of 5 mins) which as per documents should prevent this from happening. Could someone please tell me what are we doing wrong and is this the way it is suppose to work?
My second question is, we found out this is not the case when using ELB is used to balance load based on tcp connections. In this case when we scale down old tcp connections are maintained until it gets closed or times-out and new connections are routed to other instances. This is the current setup that we are using. So my question is why ELB behaves differently in both the cases and is there any way to make ELB use HTTP sticky session and drain based on tcp connection?
If you do have an answer please share with config details. Thanks.
Regarding question #1 - this is the expected behaviour of ELB Connection Draining.
From the doc : "Connection draining causes the ELB load balancer to stop sending new requests to a deregistering instance or an unhealthy instance, while keeping the existing connections open. This allows the load balancer to complete in-flight requests made to the deregistering or unhealthy instances." http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/TerminologyandKeyConcepts.html#conn-drain
ELB does not know about your session state as this is something managed by your application. Connection draining only applies at network level. If there is no open connection to your instance, ELB might elect your instance for deregistering from AutoScaling (and auto scaling will later terminate it)
Question #2 : When using TCP mode on the load balancer, it doe snot try to understand the HTTP content (such as cookie) and happily send anything received from the client to backends. The behaviour you are experimenting might be related to the way the client browser manage connections (i.e. keeps open connection). This has to be verified with a web development tool.
For your use case, I would investigate auto-scaling life cycle events to trigger a piece of code of you when auto scaling will take action on your instance. Using life cycle events, you have the occasion to trigger custom code at key life cycle states of your instance and to take additional actions or ask Auto Scaling to ABANDON the change.
Details are available here http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AutoScalingGroupLifecycle.html http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/lifecycle-hook-considerations.html