How do you load balance Azure Cloud Services WebRoles when loading a cache

Question

Given a deployment to an Azure Cloud Services WebRoles (2) using Azure SDK 3.0 on .net 4.5.2 and OS Family "4" (Windows 2012).

When the web application starts, we want to load a cache (from blob storage) that takes around 10 minutes (We have looked into moving this but currently can't)

Then when the IIS application pools recycles, we want the site to stay up.

Currently the default IIS settings with Cloud Services are:

• Not to start on load (autoStart / startMode)
• To idle every 20 minutes (idleTimeout)
• To recycle every 29 hours (periodicRestart)
• to have failures as HTTP 503s (loadBalancerCapabilities)

Because we default to 2 WebHost, we want to recycle the app pool at different times. We ideally want existing connection from the site to be redirected if one of the webhosts is loading the cache.

So far, we have a Start up task script to reconfigure the IIS AppPools

appcmd set config -section:system.applicationHost/applicationPools 

with

  /applicationPoolDefaults.autoStart:"True"
  /applicationPoolDefaults.startMode:"AlwaysRunning"
  /applicationPoolDefaults.processModel.idleTimeout:"00:00:00" 
  /applicationPoolDefaults.recycling.logEventOnRecycle:"Time,Requests,Schedule,Memory,IsapiUnhealthy,OnDemand,ConfigChange,PrivateMemory"
  /applicationPoolDefaults.recycling.periodicRestart.time:"00:00:00" 
  /~"applicationPoolDefaults.recycling.periodicRestart.schedule" 
  /+"applicationPoolDefaults.recycling.periodicRestart.schedule.[value='06:00:00']" 
  /applicationPoolDefaults.failure.loadBalancerCapabilities:"TcpLevel" 

e.g

%windir%\system32\inetsrv\appcmd set config -section:applicationPools /applicationPoolDefaults.autoStart:"True" /commit:apphost

As for code, we have looked at using a Busy flag until the cache has loaded. This doesn't appear to re-route the traffic

RoleEnvironment.StatusCheck += WebRoleEnvironment_StatusCheck;

with

        if (Busy)
        {
            e.SetBusy();
        }

The draw back is this is done in the Application_Start due to the containers that are required. I think it would be too hard to move the LoadCache() into the OnStart() of the RoleEntryPoint.

Note; We also have "Keep-alive" on by default.

Questions;

1. How do we take a WebHost offline while it loads the cache?
2. Should we change the IIS settings? https://azure.microsoft.com/en-gb/blog/iis-reset-on-windows-azure-web-role/
3. Should we use IIS 8.0 Application Initialization? http://fabriccontroller.net/iis-8-0-application-initialization-module-in-a-windows-azure-web-role/
4. What should loadBalancerCapabilities be set to? https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/failure
5. Should we try to stagger recycles? What about when we scale (add more instances) Does azure prevent that role instances are recycled at the same time?

Answer 1

This is what we have ended up with:

EDIT: Changed to a HttpWebRequest so redirects are supported

a) When a VM is deployed / OS patched we poll the httpsIn endpoint within the OnStart()

public class WebRole : RoleEntryPoint
{
    public override bool OnStart()
    {
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

        // Note: the Web Requests all run in IIS, not from this process.
        // So, we aren't disabling certs globally, just for checks against our own endpoint.
        ServicePointManager.ServerCertificateValidationCallback += (o, certificate, chain, errors) => true;

        var address = GetAddress("httpIn");

        var request = (HttpWebRequest)WebRequest.Create(address);
        request.MaximumAutomaticRedirections = 1;
        request.AllowAutoRedirect = false;
        var response = request.GetResponse() as HttpWebResponse;
        //_logger.WriteEventLog($"Response: '{response?.StatusCode}'");
        return base.OnStart();
    }

    static Uri GetAddress(string endpointName)
    {
        var endpoint = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints[endpointName];
        var address = $"{endpoint.Protocol}://{endpoint.IPEndpoint.Address}:{endpoint.IPEndpoint.Port}";
        return new Uri(address);
    }
}

b) For AppPool Recycles, we report Busy in the Global.asax

public class RoleEnvironmentReadyCheck
{
    bool _isBusy = true;

    public RoleEnvironmentReadyCheck()
    {
        RoleEnvironment.StatusCheck += RoleEnvironment_StatusCheck;
    }

    void RoleEnvironment_StatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
    {
        if (_isBusy)
        {
            e.SetBusy();
        }
    }

    public void SetReady()
    {
        _isBusy = false;
    }
}

public class WebApiApplication : HttpApplication
{
    protected void Application_Start()
    {
        var roleStatusCheck = new RoleEnvironmentReadyCheck();
        //SuperLoadCache()
        roleStatusCheck.SetReady();
    }
}

c) For the AppPool recycles, we select a time of day (03:00AM) and stagger the Roles by 30mins and stop the idle timeout in a PowerShell Script ConfigureIIS.ps1

$InstanceId = $env:INSTANCEID
$role = ($InstanceId -split '_')[-1]
$roleId = [int]$role
$gapInMinutes = 30
$startTime = New-TimeSpan -Hours 3
$offset = New-TimeSpan -Minutes ($gapInMinutes * $roleId)
$time = $startTime + $offset
$timeInDay = "{0:hh\:mm\:ss}" -f $time

Write-Host "ConfigureIIS with role: $role to $timeInDay"

& $env:windir\system32\inetsrv\appcmd set config -section:system.applicationHost/applicationPools /applicationPoolDefaults.processModel.idleTimeout:"00:00:00" /commit:apphost
& $env:windir\system32\inetsrv\appcmd set config -section:system.applicationHost/applicationPools /applicationPoolDefaults.recycling.logEventOnRecycle:"Time,Requests,Schedule,Memory,IsapiUnhealthy,OnDemand,ConfigChange,PrivateMemory" /commit:apphost
& $env:windir\system32\inetsrv\appcmd set config -section:system.applicationHost/applicationPools /applicationPoolDefaults.recycling.periodicRestart.time:"00:00:00" /commit:apphost
& $env:windir\system32\inetsrv\appcmd set config -section:system.applicationHost/applicationPools /~"applicationPoolDefaults.recycling.periodicRestart.schedule" /commit:apphost
& $env:windir\system32\inetsrv\appcmd set config -section:system.applicationHost/applicationPools /+"applicationPoolDefaults.recycling.periodicRestart.schedule.[value='$timeInDay']" /commit:apphost

And pass the RoleId to the ConfigureIIS.cmd

PowerShell -ExecutionPolicy Unrestricted .\ConfigureIIS.ps1 >> "%TEMP%\StartupLog.txt" 2>&1

EXIT /B 0

Set within the ServiceDefinition.csdef

 <Task commandLine="ConfigureIIS.cmd" executionContext="elevated" taskType="simple">
    <Environment>
      <Variable name="INSTANCEID">
        <RoleInstanceValue xpath="/RoleEnvironment/CurrentInstance/@id"/>
      </Variable>
    </Environment>
  </Task>