Azure Cloud Service : Scaled in web site (web role) instance recieves requests before iis is ready

Question

Description

I got a web role hosted in a cloud service that is a web site. There are predictible load spikes that we handle with a custom monitoring (a worker role) that will periodically scale in or scale out depending on the predicted load profile. That part is working.

Issue

After a new instance of the web site is mounted by the monitoring, we just had a case of a user whose request reach that new instance before the new instance was fully load. It resulted in an unaccessible web site blank page on the web browser. Given our typical target client it is very bad. This happen when a tester connected (for the first time) to the web site approximately during the scale in process.

Do my question is, is there a way to ensure that a web role has fully loaded IIS7, started the apppools and finished all the web site warmup process before any request is directed to it? Or in a more general way, can we manualy decide wich instance is available to the load balancer ? by that I mean that I assume there is a built-in load balancer over a cloud service automaticaly distributing requests.

Thanks in advance.

Answer 1

See https://learn.microsoft.com/en-us/archive/blogs/kwill/role-instance-restarts-due-to-os-upgrades, specifically the Common Issues #5 and associated code.

If your website takes several minutes to warmup (either standard IIS/ASP.NET warmup of precompilation and module loading, or warming up a cache or other app specific tasks) then your clients may experience an outage or random timeouts. After a role instance restarts and your OnStart code completes then your role instance will be put back in the load balancer rotation and will begin receiving incoming requests. If your website is still warming up then all of those incoming requests will queue up and time out. If you only have 2 instances of your web role then IN_0, which is still warming up, will be taking 100% of the incoming requests while IN_1 is being restarted for the Guest OS update. This can lead to a complete outage of your service until your website is finished warming up on both instances. It is recommended to keep your instance in OnStart, which will keep it in the Busy state where it won't receive incoming requests from the load balancer, until your warmup is complete. You can use the following code to accomplish this:

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        // For information on handling configuration changes
        // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
        IPHostEntry ipEntry = Dns.GetHostEntry(Dns.GetHostName());
        string ip = null;
        foreach (IPAddress ipaddress in ipEntry.AddressList)
        {
            if (ipaddress.AddressFamily.ToString() == "InterNetwork")
            {
                ip = ipaddress.ToString();
            }
        }

        string urlToPing = "http://" + ip;
        HttpWebRequest req = HttpWebRequest.Create(urlToPing) as HttpWebRequest;
        WebResponse resp = req.GetResponse();
        return base.OnStart();
    }
}