Retained size calculation including stack frame variables?

Question

I've found a ton of questions on "retained size" and the accepted answer seems to be:

The retained size for an object is the quantity of memory this objects preserves from garbage collection.

Now, I've been working on programmatic computation of the retained size in a hprof file (as defined here), using the Netbeans profiler library (the retained size calculation is done in HprofHeap.java). Works just fine (sorry, used kotlin for brevity):

val heap: Heap = HeapFactory.createHeap(myHeap.toFile())
val threadClass: JavaClass = heap.getJavaClassByName("java.lang.Thread")
val instanceFilter = { it: Instance -> threadClass == it.getJavaClass() }
val sizeMap = heap.allInstances
            .filter { instanceFilter(it) }
            .toMap({ findThreadName(it) /* not shown */ }, { it.retainedSize })

What I noticed when the sizeMap had only marginal numbers of retained sizes is that Netbeans computes retained sizes only for objects that are not on the stack. So local variables (allocated on the stack) assigned to the Thread would not be included in the retained size.

My question is: is there a way to make the netbeans library consider the stack elements as dependent objects the way for example the Yourkit Profiler does it's calculation? How would I go about adding such a feature if the answer to the previous question is "no"?

Answer 1

A bit of digging found that the JVM heap dumper creates an entry of type ROOT JAVA FRAME for a stack local variable (compare VM_HeapDumper::do_thread). Since I can grep for that in the heap, here's what I did:

val threadClass: JavaClass = heap.getJavaClassByName("java.lang.Thread")

val keyTransformer = { it: Instance -> findThreadName(it) }
val instanceFilter = { it: Instance -> it.getJavaClass() == threadClass }

val stackLocals = heap.gcRoots
        .filter { it.kind == GCRoot.JAVA_FRAME }
        .groupBy { (it as JavaFrameGCRoot).threadGCRoot }

val sizeMap = heap.allInstances
      .filter { instanceFilter(it) }
      .toMap(
          { keyTransformer(it) }, 
          { 
              val locals = stackLocals[heap.getGCRoot(it)]
              val localSize = locals!!.sumBy { it.instance.retainedSize.toInt() }
              it.retainedSize + localSize
          })

return Report(
        sizeMap.values.sum(),
        sizeMap.keys.size.toLong(),
        sizeMap.maxBy { it.value }?.let { it.toPair() } ?: ("n/a" to 0L))

This solution is based on finding the GC root for each thread (should be the Thread itself), then sort to the stored gc root of the JAVA FRAME (the thread [= GC root] id is part of the stored entry data).

There's still a slight difference compared to the values from Yourkit, probably due to me missing ROOT JNI LOCAL entities, but it's close enough for me.