Classic ASP: response time periodically slows down extremely
I have a problem with a bunch (around 50) of classic ASP-Sites on Win2012R2 with Access-Databases, which drives us crazy.
All asp-pages of all sites on this server run smoothly for around 45 seconds, after that period they (all) completely stop responding to any click for 15 to 20 seconds, then this delay disappears again for the next 45 seconds like it never existed before, it re-appears again - and so on. This effect started out of nothing a few weeks ago, after several months without any problems.
Static HTML-pages are not affected, and it seems, even asp-pages without connecting to their database run fine. We, therefore, tried testing to convert from Access to SQLExpress, but this didn't change anything - even the converted site was affected in the same way (so it seems not to be Access).
We then tried to stop all sites in IIS and re-activating just one single site with very few visitors to see if it only appears, when many requests are sent to the server. But the effect still showed up, even after Restarting IIS and even after restarting the whole machine with just one website activated in IIS. It seems to be completely independent from the number of effects, just like the server (rather: the asp-engine of IIS) being busy with itself in a periodical pattern.
What we can see in performace monitor (see screenshot): while requests/sec goes down to 0 at some moment, when the effect starts, the number of requests executed continuosly accumultes from a normal level (which looks "logical" to me, but only describes the effect, not justifies, where it comes from). A few seconds before the effect vanishes, request/sec again grow and these counters revert to normal values.
We had a similar problem a year ago on a Windows 2008-Server, where the sites ran without any problems for several years and then it started out of nothing. After testing some of the sites on a server of another hoster, we found out, the problems didn't appear on his server with Windows 2012 R2 (and still don't for a full year, while hosting 3 of our sites there). At another hosters virtual Windows 2012R2-Server we have another single site hosted with more traffic than most of our others and even there the problem didn't appear since a full year now. So we our hoster switched over to WinServer2012R2 and - bingo - all the problems were gone. All sites performed like a charm again from that moment on without changing anything but the OS.
We then stopped investigating the issue, thinking the problem relates to the OS. But around 9 months later, it re-appeared and after hours and hours of investigating we have no idea, what to search for and what to do (beside of moving all our sites to the other hosters server, which isn't a real solution to the problem and we cannot guarantee, the effect will not re-appear on this machine sometimes in the future).
I definitively found a solution by myself, but in a totally random way. After weeks of searching for a solution to the problem, I worked on cleaning up the server's hard disk and deleted all files in Windows/temp-folder (> 18.000 files!). And since this moment (4 days ago), the described response lag never showed up any more! But a small bunch of new .tmp-files were created in the folder.
My theory is: maybe every time a user visits one of the websites (which opens connections to its Access database, causing a .ldb-file in the database folder), a randomly(?) named .tmp-file (like: jet12f0.tmp) is created in Windows/temp folder in parallel. These files are "normally" deleted again, as database connection closes and the .ldb disappears. Maybe some of the connections are not closed correctly, therefore the corresponding .tmp-file in the Windows/temp folder resides there as an "orphan", literally forever. As time goes by, the folder fills up with these orphaned files. And then it comes, that a new .tmp-file should be generated, but with a name of a still existing "orphaned" .tmp-file. This now causes the server to stop all actions, because it is not possible to establish the new file, named like an existing. After 15 to 20 seconds the conflict is solved by some mechanism (unknown to me) and all runs perfect again, until the next conflict arises around 45 seconds later. And so on...
I must assume: this is only a "amateur" theory, I'm not a server "Guru".
Cleaning up this temp-folder from time to time seems to prevent the server from getting into this situation, because there are no file/naming conflicts.
I agree: The real solution would be finding the problem in the code (if there is one), but we can live with that situation, comparing the effort to find the problem with just cleaning up the temp-folder once in a month or so ;-)