DTrace missing Java frames with ustack(). Running on Joyent SmartOS infrastructure container

Question

I cannot get any Java stack with dtrace in a Joyent SmartOS instance.

I tried the java:15.1.1 image and a plain SmartOS 'base64' image, where I installed openjdk 8.

I most basic example: cat Loop.java

[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# cat Loop.java 
class Loop {

    public static void main(String[] args) throws InterruptedException {
        while (true) {
                System.out.println("Sleepy");
                Thread.sleep(2000);
        }
    }
}
[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# javac Loop.java 
[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# java Loop

I added the libdtrace_forceload.so as recommended.

export LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so

It is a 64 bit JVM:

[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 /demo]# java -version
openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-pkgsrc_2015_05_29_19_05-b00)
OpenJDK 64-Bit Server VM (build 24.76-b04, mixed mode)

When I run dtrace, and use jstack, I get the C-stacks. However, the JAVA frames are the raw addresses, quite useless:

[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# pgrep -fn "java Loop"
32597
[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# dtrace -n 'syscall:::entry/pid == 32597/ { @num[ustack(20)] = count(); }'
dtrace: description 'syscall:::entry' matched 237 probes
^C


              libc.so.1`__write+0xa
              libjvm.so`_ZN2os5writeEiPKvj+0x128
              libjvm.so`JVM_Write+0x34
              libjava.so`writeBytes+0x1b5
              libjava.so`Java_java_io_FileOutputStream_writeBytes+0x1f
              0xffffbf7ffa612d98
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa606058
              0xffffbf7ffa6004e7
              libjvm.so`_ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread+0x31d

*snip*

I do see hotspot probes are available:

[root@7e8c2a25-c852-4967-b60c-7b4fbd9a1de5 ~]# dtrace -l | grep hotspot | more
 6103 hotspot32597         libjvm.so     _ZN17VM_GenCollectFull4doitEv gc-begin
 6104 hotspot32597         libjvm.so _ZN15VM_GC_Operation13notify_gc_endEv gc-end
 6105 hotspot32597         libjvm.so _ZN26VM_GenCollectForAllocation4doitEv gc-end
 6106 hotspot32597         libjvm.so _ZN35VM_GenCollectForPermanentAllocation4doitEv gc-end
 6107 hotspot32597         libjvm.so     _ZN17VM_GenCollectFull4doitEv gc-end
 6132 hotspot32597         libjvm.so _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread class-initialization-end
 6133 hotspot32597         libjvm.so _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread class-initialization-erroneous
 6441 hotspot_jni32597         libjvm.so                jni_DeleteLocalRef DeleteLocalRef-entry
 6442 hotspot_jni32597         libjvm.so                jni_DeleteLocalRef DeleteLocalRef-return
 6443 hotspot_jni32597         libjvm.so           jni_DeleteWeakGlobalRef DeleteWeakGlobalRef-entry
 6444 hotspot_jni32597         libjvm.so           jni_DeleteWeakGlobalRef DeleteWeakGlobalRef-return
 6445 hotspot_jni32597         libjvm.so                 jni_DestroyJavaVM DestroyJavaVM-entry
 6446 hotspot_jni32597         libjvm.so                 jni_DestroyJavaVM DestroyJavaVM-return

Question: Is there a way to list ustack helpers and if they are loaded? Any way to get the Java stack?

Answer 1

In your example with untranslated Java stack frames, you appear to be using the ustack() action. In order to get translated frames, I believe you ought to be using the jstack() action instead.