Search code examples
cachingload-balancingibm-cloud-infrastructure

Softlayer's Local Load Balancing - HTTPS requests not being distributed

I'm using Softlayer's Local Load Balancing: https://www.ibm.com/cloud-computing/bluemix/load-balancing

The current setup is two servers with 50/50 split for both HTTP and HTTPS traffic. The method is Round Robin with no persistency setting.

For HTTP traffic the requests are correctly alternating between the two servers. Changes in load balancer are picked up instantly.

The issue is with HTTPS:

  • For HTTPs traffic only one server gets hit - it's chosen at random and all the subsequent requests stick with it and never go to the other server.
  • The requests continue hitting the same server even if it's disabled in the load balancer. This behavior is going on forever.
  • If I stop making the requests and wait for about a minute and resume, the other enabled server gets picked up correctly.

I put the test log at the bottom of this question to demonstrate the behavior. It contains the server name and a generated timestamp (to ensure pages aren't being cached).

Now here comes the tricky part. In Firefox the behavior is the same as described but when I press CTRL+F5 I get to the other server. This would imply something is cached, but not the pages themselves (as demonstrated with the timestamp). I did not see anything suspicious in the headers or cookies. Plus I'm not sure what sort of header would make the LB redirect traffic to a single server.

Firewall was disabled while performing this test. Tracing the requests ends on the load balancer (the actual server's IP is never given out to the client).

LOAD BALANCER SETUP

Load Balancer settings

HEADERS

GET https://test.devcorp.kangogift.com/login?ReturnUrl=%2F HTTP/1.1 Host: test.devcorp.kangogift.com Connection: keep-alive Cache-Control: max-age=0 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8 Accept-Encoding: gzip, deflate, sdch, br Accept-Language: cs,en;q=0.8,en-US;q=0.6 Cookie: ASP.NET_SessionId=dyvydeine15qe5wj5yikhmkf; __utma=212613646.1973225437.1495466549.1495570138.1495575175.4; __utmc=212613646; __utmz=212613646.1495466549.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); .AspNet.Cookie.v1=51ToSLNxI2eb0p1v106S0-qjXIEJd7SMmZrr9KV2UPcSvGuLatyM2UBWbP3Bh42C0NDZXpw5zjnNXHvXtzY1ju2mxihZOG25oQMvmK5A6oPwZiG8sq2bKp3iYAiZ3PEEo79XrOBfoPl3G0s0RAwLWCThqv6dYw7OBEZ-cpKuxKnRnpOH1lCDrnHj73BEmMop-GVVNxkIFbVhbEw16HZWWiNQC1EHBu9bl2yOEG8EI9z-gvVRmRk1llUEi5y9muivk8lwhatSlT4t64YAsnBhvCLVYTkqfJA41G9yDIzLIh1zTIMrgEbCAGsleFaWtcD8zX0KJeM2N4-NwQ9hxRRlK16IHKcOChx5XlJmQL0yd5aDIogjjCejYCWkhs_aM-H6HrGM0CpW1erIPDi7AIwUTlkZP4ZVI0OjCnmwEJ2fqmyL1JkGGDkJHi4sFUGcxxKYvtryJSPloOvuXBhWUVQ6ew X-CookiesOK: I explicitly accept all cookies

HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Content-Encoding: gzip Vary: Accept-Encoding Server: Microsoft-IIS/10.0 X-AspNetMvc-Version: 5.2 X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET P3P: policyref="http://www.****.com/privacy",CP="CURa ADMa DEVa TAIa CONa OUR DELa BUS IND PHY ONL UNI PUR COM NAV DEM STA" Date: Fri, 26 May 2017 19:59:29 GMT Content-Length: 7405

TEST RESULT:

---- HTTP test 20 times ----

Server: SERVER2, Time: 5/26/2017 11:30:47 AM Server: SERVER1, Time: 5/26/2017 11:30:33 AM Server: SERVER2, Time: 5/26/2017 11:30:49 AM Server: SERVER1, Time: 5/26/2017 11:30:34 AM Server: SERVER2, Time: 5/26/2017 11:30:50 AM Server: SERVER1, Time: 5/26/2017 11:30:35 AM Server: SERVER2, Time: 5/26/2017 11:30:51 AM Server: SERVER1, Time: 5/26/2017 11:30:37 AM Server: SERVER2, Time: 5/26/2017 11:30:53 AM Server: SERVER1, Time: 5/26/2017 11:30:38 AM Server: SERVER2, Time: 5/26/2017 11:30:54 AM Server: SERVER1, Time: 5/26/2017 11:30:39 AM Server: SERVER2, Time: 5/26/2017 11:30:55 AM Server: SERVER1, Time: 5/26/2017 11:30:40 AM Server: SERVER2, Time: 5/26/2017 11:30:56 AM Server: SERVER1, Time: 5/26/2017 11:30:42 AM Server: SERVER2, Time: 5/26/2017 11:30:58 AM Server: SERVER1, Time: 5/26/2017 11:30:43 AM Server: SERVER2, Time: 5/26/2017 11:30:59 AM Server: SERVER1, Time: 5/26/2017 11:30:44 AM Server: SERVER2, Time: 5/26/2017 11:31:00 AM

---Disabling Server SERVER1 on LB---

----HTTP test 20 times----

Server: SERVER2, Time: 5/26/2017 11:43:39 AM Server: SERVER2, Time: 5/26/2017 11:43:40 AM Server: SERVER2, Time: 5/26/2017 11:43:41 AM Server: SERVER2, Time: 5/26/2017 11:43:41 AM Server: SERVER2, Time: 5/26/2017 11:43:42 AM Server: SERVER2, Time: 5/26/2017 11:43:43 AM Server: SERVER2, Time: 5/26/2017 11:43:43 AM Server: SERVER2, Time: 5/26/2017 11:43:44 AM Server: SERVER2, Time: 5/26/2017 11:43:45 AM Server: SERVER2, Time: 5/26/2017 11:43:45 AM Server: SERVER2, Time: 5/26/2017 11:43:46 AM Server: SERVER2, Time: 5/26/2017 11:43:46 AM Server: SERVER2, Time: 5/26/2017 11:43:47 AM Server: SERVER2, Time: 5/26/2017 11:43:48 AM Server: SERVER2, Time: 5/26/2017 11:43:48 AM Server: SERVER2, Time: 5/26/2017 11:43:49 AM Server: SERVER2, Time: 5/26/2017 11:43:50 AM Server: SERVER2, Time: 5/26/2017 11:43:50 AM Server: SERVER2, Time: 5/26/2017 11:43:51 AM Server: SERVER2, Time: 5/26/2017 11:43:52 AM Server: SERVER2, Time: 5/26/2017 11:43:52 AM

---Both servers enabled on LB---

---- HTTPS test 20 times ----

Server: SERVER1, Time: 5/26/2017 11:34:22 AM Server: SERVER1, Time: 5/26/2017 11:34:23 AM Server: SERVER1, Time: 5/26/2017 11:34:24 AM Server: SERVER1, Time: 5/26/2017 11:34:24 AM Server: SERVER1, Time: 5/26/2017 11:34:25 AM Server: SERVER1, Time: 5/26/2017 11:34:25 AM Server: SERVER1, Time: 5/26/2017 11:34:26 AM Server: SERVER1, Time: 5/26/2017 11:34:27 AM Server: SERVER1, Time: 5/26/2017 11:34:27 AM Server: SERVER1, Time: 5/26/2017 11:34:28 AM Server: SERVER1, Time: 5/26/2017 11:34:29 AM Server: SERVER1, Time: 5/26/2017 11:34:29 AM Server: SERVER1, Time: 5/26/2017 11:34:30 AM Server: SERVER1, Time: 5/26/2017 11:34:30 AM Server: SERVER1, Time: 5/26/2017 11:34:31 AM Server: SERVER1, Time: 5/26/2017 11:34:32 AM Server: SERVER1, Time: 5/26/2017 11:34:32 AM Server: SERVER1, Time: 5/26/2017 11:34:33 AM Server: SERVER1, Time: 5/26/2017 11:34:34 AM Server: SERVER1, Time: 5/26/2017 11:34:34 AM Server: SERVER1, Time: 5/26/2017 11:34:35 AM

---Disabling Server SERVER1 on LB---

----HTTPS test 20 times----

Server: SERVER1, Time: 5/26/2017 11:35:01 AM Server: SERVER1, Time: 5/26/2017 11:35:01 AM Server: SERVER1, Time: 5/26/2017 11:35:02 AM Server: SERVER1, Time: 5/26/2017 11:35:03 AM Server: SERVER1, Time: 5/26/2017 11:35:03 AM Server: SERVER1, Time: 5/26/2017 11:35:04 AM Server: SERVER1, Time: 5/26/2017 11:35:05 AM Server: SERVER1, Time: 5/26/2017 11:35:05 AM Server: SERVER1, Time: 5/26/2017 11:35:06 AM Server: SERVER1, Time: 5/26/2017 11:35:06 AM Server: SERVER1, Time: 5/26/2017 11:35:07 AM Server: SERVER1, Time: 5/26/2017 11:35:08 AM Server: SERVER1, Time: 5/26/2017 11:35:08 AM Server: SERVER1, Time: 5/26/2017 11:35:09 AM Server: SERVER1, Time: 5/26/2017 11:35:10 AM Server: SERVER1, Time: 5/26/2017 11:35:10 AM Server: SERVER1, Time: 5/26/2017 11:35:11 AM Server: SERVER1, Time: 5/26/2017 11:35:12 AM Server: SERVER1, Time: 5/26/2017 11:35:12 AM Server: SERVER1, Time: 5/26/2017 11:35:13 AM Server: SERVER1, Time: 5/26/2017 11:35:13 AM

----Delay 1 Minute----

----HTTPS test 20 times----

Server: SERVER2, Time: 5/26/2017 11:36:32 AM Server: SERVER2, Time: 5/26/2017 11:36:33 AM Server: SERVER2, Time: 5/26/2017 11:36:34 AM Server: SERVER2, Time: 5/26/2017 11:36:34 AM Server: SERVER2, Time: 5/26/2017 11:36:35 AM Server: SERVER2, Time: 5/26/2017 11:36:36 AM Server: SERVER2, Time: 5/26/2017 11:36:36 AM Server: SERVER2, Time: 5/26/2017 11:36:37 AM Server: SERVER2, Time: 5/26/2017 11:36:37 AM Server: SERVER2, Time: 5/26/2017 11:36:38 AM Server: SERVER2, Time: 5/26/2017 11:36:39 AM Server: SERVER2, Time: 5/26/2017 11:36:39 AM Server: SERVER2, Time: 5/26/2017 11:36:40 AM Server: SERVER2, Time: 5/26/2017 11:36:41 AM Server: SERVER2, Time: 5/26/2017 11:36:41 AM Server: SERVER2, Time: 5/26/2017 11:36:42 AM Server: SERVER2, Time: 5/26/2017 11:36:43 AM Server: SERVER2, Time: 5/26/2017 11:36:43 AM Server: SERVER2, Time: 5/26/2017 11:36:44 AM Server: SERVER2, Time: 5/26/2017 11:36:44 AM Server: SERVER2, Time: 5/26/2017 11:36:45 AM

Solution

  • As a hunch, likely because HTTPS maintains some state, so for performance reasons perhaps it stays on one server, or the load balancer isn't too smart and just sees a persistent TCP connection in port 443 and maintains/never disconnects it (until you idle timeout he connection by waiting at your browser for a few minutes).