ASM Bytecode replacement function not completing

Question

I have a slightly complicated architecture in which I am trying to instrument a class with the JVMTI function RetransformClasses, and event hook ClassLoadFileHook, while using JNI to go up to Java space to perform the transformation. In ClassLoadFileHook, I check if it's being fired for the class I want to instrument, and if so, I make a JNI call to a Java class where I perform the instrumentation using ASM 5.0.4. That is, in Java space I get a byte array representing the instrumented class, and then call back down to my native code to apply the update.

The problem I am experiencing is that in my Java code, where I retransform the class, the code is hanging during/after I instantiate the ClassReader:

/**
 * Transform some class and then call back down to native code to apply
 * the transformation.
 *
 * @param existingClass The existing class to change.
 */
public synchronized void transformSomeClass(byte[] existingClass) {
    System.out.println("Transformation function started. Existing class len: " + existingClass.length);
    try {
        ClassReader cr = new ClassReader(existingClass);
        // No statements below here are being executed.
        // I'm also not sure if `new ClassReader` is being successfully
        // executed.
        System.out.println("After ClassReader instantiation");
        ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES);
        System.out.println("After ClassWriter instantiation");
        ClassVisitor cv =
                new MyCustomClassAdapter(cw, this.lineNumberToPlaceUpdate_);
        System.out.println("After ClassVisitor instantiation");
        cr.accept(cv, 0);
        System.out.println("After cv accept.");

        System.out.println("Applying class transformation natively.");
        // Call back down to native code to apply the update in ClassLoadFileHook.
        applyClassTransformNative(cw.toByteArray());
        System.out.println("Transformation function ended.");
    } catch (Exception e) {
        // This is never being fired.
        System.err.println("Class transformation could not be completed. Error message: " + e.getMessage());
        e.printStackTrace();
    } finally {
        // This print statement is never executed.
        System.err.println("Going back to native code from finally");
    }
}

As I mention in the comments of the code, nothing after the ClassReader intantiation statement is ever executed. Even the finally block is not executed. I'm not sure what could be going on here -- why would instantiating a ClassReader that takes a generic array of bytes cause the code to hang like this? I haven't actually performed any operations on the class.

Answer 1

If your calling into the Java code causes class loading, you implicitly synchronize on the class loader that loads a class. I assume that you trigger such a class loading on the native layer, and then get blocked when you try to trigger it again on the Java layer. This is why you never progress beyond the first ASM class when you try to transform the class in Java.