After perusing stack exchange & numerous MSDN articles for months, I'm coming to the community looking for help. I'm the IIS Admin for my organization & I've been seeing an issue wherein when a change is made to a .NET application, the page takes a very long time to load when a request is first submitted to the page. When I say a long time, I'm saying anywhere from five minutes to a half hour at times. I've tried a number of items to fix the issue. But it's my belief that there's a configuration setting buried somewhere that is causing the issue. Once the page loads once, it's load time is normal going forward. It seems to happen regardless of the application, although some take longer than others. It's also not consistent. Sometimes it'll take seven minutes to load after a change. That same application may take seventeen minutes to load the next time a change is made... The changes are usually very minor, a new image box is moved in, or a new link added. Nothing major.
If an app takes twenty seconds to a minute to load the first time, we're not concerned. But the ten, fifteen minute, sometimes half hour load time is what is not acceptable.
The issue happens regardless of whether or not it's a static content application, or if it's reaching out to a data source. Any connections to a data source are configured at the app level. We only utilize that on a handful of apps & I verified the connection information is valid in the web.config for those apps that do reach out to a data source. We're using Windows Authentication on every app.
We run a three tiered environment, all running Windows Server 2012 R2 Standard with 16gb of ram & a multicore CPU setup. We're running IIS 8.5 on .NET 4.0.30319. The app pool is utilizing the integrated pipeline with support for 32-bit apps (which the apps are). These are VMware Hosts. The server is rebooted once a week. The issue does not occur on our test or dev servers. Only our production server. It doesn't matter what time of day the change is made.
In December of 2016, I ported over all of our .NET applications from an older Windows 2003 system running IIS 6.0 to the new Windows 2012 system. While we worked with the developer to alter any hardcoded hostnames in the apps, we ultimately had to install a CNAME host record to re-direct the old hostname to the new one. The issue seems to have started around this time.
One issue I noticed was that the developer was compiling all apps in debug mode. We changed this setting to false, but it only marginally helped in some cases.
I tried the following as well:
Segregated a given app & set the app pool to always run. Also tried changing the application pool identity to run as a network service, or as a service account user.
Adding the application initialization role & configuring in IIS to always run the app pool & enabled preload at the app level. - I backed this out when I saw it made no change.
I married up all the roles between our test/dev servers where the issue doesn't happen with that of production.
Disabled idle timeout at the app pool level.
Compared settings between test & production regarding compilation settings, time-outs, etc
None of these changes made any difference. I can't find anything differentiating the test/dev boxes where there is no issue & the production problematic server. Please let me know what additional information you may need & I appreciate the help in advance!!
Thanks, Mike
We found the problem. Our AV software was the culprit. Specifically Trend Micro Deep Security.
When a request was made & the program was going to compile in the temp area (C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files) - I noticed it seemed to be taking a very long time for the files to build out. Pre-compiling saw the same behavior. I asked our LAN admin to add an exception for this particular folder. Once that was done, it's only taking 5-10 seconds on first load, which is acceptable. Thanks again for your input.