How to Mix Manual and Automatic Calculation of Max Locals, Max Stack, and Frames in ASM per Method Basis?

I'm currently working on a project where I generate Java bytecode from my own intermediate representation using the ASM library. For some methods in my code, I already have pre-calculated values for the maximum stack size, local variables, and frame information. However, for other methods, I lack this information and would prefer ASM to automatically compute these values for me.

The challenge arises because ASM's ClassWriter allows specifying computation options (COMPUTE_MAXS and COMPUTE_FRAMES) only at the class level, affecting all methods in the class. I'm looking for a way to selectively apply ASM's automatic computation on a per-method basis, such that:

For methods where I have pre-calculated max locals and stack sizes, I manually set these values.
For the rest, ASM computes the max locals, stack sizes, and frames automatically.

I attempted to extend ClassWriter but have not found a way to return a different implementation of MethodWriter. Moreover, MethodWriter is declared as a final class, preventing it from being extended.

If I set COMPUTE_FRAMES, ASM simply ignores the values that I specify using visitMaxs and calculates them from the beginning.

I appreciate any insights or alternative strategies.

Solution

Recently, the setFlags method was added to the ClassWriter. This method changes the computation strategy of method properties like max stack size, the max number of local variables, and frames. The following code demonstrates how to change computation strategies:

final ClassWriter writer = new ClassWriter(0);
writer.visitMethod(…).visitEnd() //computes nothing
writer.setFlags(ClassWriter.COMPUTE_MAXS);
writer.visitMethod(…).visitEnd() //computes maxs
writer.setFlags(ClassWriter.COMPUTE_FRAMES);
writer.visitMethod(…).visitEnd()  //computes frames

Important! It changes the behavior of only new method visitors returned from visitMethod. All the previously returned method visitors keep their previous behavior.

Old Workarounds

I haven’t found an optimal and straightforward solution for this problem yet, so here are several ways that you might apply to achieve the desired behavior:

Compute All These Values Yourself

As @holger mentioned, it might be much simpler to just compute these values in place.

More than often, just computing the values is simpler. There seems to be a widespread irrational fear of doing that but when you can express the code changes in form of a computer program you can also express the effects on the stack frame in form of a computer program. In most cases, it’s easier than you think.

So, maybe it’s the best option to follow.

Fork ASM and Modify The Required Code Yourself

As @alzs mentioned, you can fork and modify the Java ASM library for your needs. This can lead to future maintainability issues, however it’s a proper and optimal solution, but which might require extra effort to properly implement and maintain this in the future.

Reflection Workaround

Since it is still impossible to achieve using the ASM library, you can use reflection to change the ClassVisitor at runtime:

private abstract class ReflectionClassWriter extends ClassVisitor {
    private final ClassWriter writer;
    private ReflectionClassWriter(final int api, final ClassWriter writer) {
        super(api, writer);
        this.writer = writer;
    }
    /**
     * Implement this method that will decide whether we should compute frames,
     * locals, and stack values.
     *
     * @return True if we should compute frames, locals, and stack values; false otherwise.
     */
    abstract boolean shouldWeComputeFrames();
    @Override
    public MethodVisitor visitMethod(
        final int access,
        final String name,
        final String descriptor,
        final String signature,
        final String[] exceptions
    ) {
        final MethodVisitor result;
        if (this.shouldWeComputeFrames()) {
            final ClassVisitor delegate = this.getDelegate();
            try {
                final Field field = ClassWriter.class.getDeclaredField("compute");
                field.setAccessible(true);
                final int previous = field.getInt(delegate);
                field.setInt(delegate, 4);  // MethodWriter#COMPUTE_ALL_FRAMES = 4;
                final MethodVisitor original = this.visitMethod(access, name, descriptor, signature, exceptions);
                field.setInt(delegate, previous);
                result = original;
            } catch (final NoSuchFieldException | IllegalAccessException exception) {
                throw new IllegalStateException(
                    String.format(
                        "Can't set compute field for ASM ClassWriter '%s' and change the computation mode to COMPUTE_ALL_FRAMES",
                        delegate
                    ),
                    exception
                );
            }
        } else {
            result = this.visitMethod(access, name, descriptor, signature, exceptions);
        }
        return result;
    }
}

I should admit that it’s an extremely suboptimal solution and should be used only as a temporary solution. Hopefully, in future versions of the ASM library, we will be able to do it in a more straightforward way.

Use FlexibleClassVisitor

Lastly, you might try to use the FlexibleClassVisitor solution suggested by @alzs. The code was described here. I tried to use it myself, and this code failed in my context, but it might help in your case. Apparently, with some modifications.