I'm looking for indicators that I can programatically check that would hint at an impending failure of my application. I'm looking for general things like "the number of free IO threads is dangerously low", "memory available to the app pool is low", and "processor usage is high".
This is for C#/asp.net and IIS.
Examples of checking the value programatically would be nice, but not required. Any good ideas welcome.
My thoughts:
Determine the performance inputs and outputs of your webapp.
Profile your application under real request data. Build a nice table of inputs and their associated outputs. For example:
4 users calling page1 -> costs 4 - 6 mb, 4 threads
5 users calling page1 -> costs 7 - 14 mb, 5 threads
2 users calling page2 -> costs 120 - 200 mb, 1 thread
Find outputs that often cause failure and find which inputs cause those outputs. Build a nice max-likelyhood model of failure.
When your inputs start approaching your failure outputs, impending failure is likely with some probability. Record when failure does and does not occur and feed this information back into your table. Your webapp will learn when it is about to fail.
Finding the outputs is the easiest part.
See SO questions How to get memory used in c#, How to get the cpu usage in C# and for the more general question What key performance monitors should I watch for asp.net application.
Key points from those questions:
GC.GetTotalMemory - tells you how much is allocated by the Garbage Collector.
Processor Object - tells you all sorts of interesting preformance stats on the cpu (cpu idle time, usauge, etc).