what are the 'self' and 'total' columns? The 'total' column does not add up to 100% (much higher) and it looks like the self does. I suspect self is non-cumulative and total is. So if methodA calls methodB calls methodC, in self Id see the % for each method call individually, whereas in total methodA would show the total of all three methods, methodB would show the 2, and so on.
Is this correct?
Suppose you have this program:
main() calls A() calls B() calls C(), and C hangs in a loop for 10 seconds. The CPU-profiler would say something like this:
total time: 10 sec
routine self% inclusive%
main 0 100
A 0 100
B 0 100
C 100 100
The self time of C would be 10 seconds, 100%. The self time of the others would be essentially zero.
The total (inclusive) time of every one of them would be 10 seconds or 100%. You don't add those up.
On the other hand, suppose C spends its 10 seconds doing I/O. Then the CPU-only profiler would say something like this:
total time: 0 sec
routine self% inclusive%
main ? ?
A ? ?
B ? ?
C ? ?
because the only actual CPU time it uses is so short that basically no samples hit it, so to get the percents it is dividing by zero.
OTOH if the samples were on wall-clock time, you would get the first output.
A better type of profiler is one that samples the call stack, on wall clock time and tells you inclusive time as a percent of total, and gives it to you at the line-of-code level, not just for functions. That's useful because it's a direct measure of how much could be saved if the line were executed less, and almost no problem can hide from it. Examples of such profilers are Zoom and LTProf, and I'm told OProfile can do it. There's a simple method that works with any language and requires only a debugger.