How do you build a AppDynamic or New Relic kind of system that collects performance metrics of your application including detailed call tree stats by merely installing a software on the servers where your application runs?
Is it even possible to collect such metrics without compiling your apps with debug information?
What are the performance trade offs to consider when building such a service? How do such software minimize the performance impact they themselves might be having on the application.
The way that these products generally work is by doing bytecode injection / function interposition / monkey-patching on commonly used libraries and methods. For instance, you might hook into JDBC query methods, servlet base classes, and HTTP client libraries. When a request enters the application, track all the important methods/calls it makes, and log them in some way. Take the data and crunch it into analytics, charts, and alerts.
On top of that, you can start to add in statistical profiling or other options.
The tricky things are tracking requests across process boundaries and dealing with the volume of performance data you'll gather. (I work on this problem at AppNeta)
One thing to check out is Twitter Zipkin (https://github.com/twitter/zipkin), doesn't support much and pretty early-stage but interesting project.