how to avoid recursive calls with byte buddy - java.lang.StackOverflowError

Question

I have an advice which calls a similar method in the advice. How do we make sure the advice gets called once and only once. Right now as the method I am calling within advice is the same as the one being instrumented, it goes into recursive calling and results in java.lang.StackOverflowError.

 transform(
              new AgentBuilder.Transformer.ForAdvice()
.include(JettyHandlerAdvice.class.getClassLoader())
.advice(named("addFilterWithMapping").and(ElementMatchers.takesArgument(0,named("org.eclipse.jetty.servlet.FilterHolder"))),JettyHandlerAdvice.class.getName())
                        )

Advice

@Advice.OnMethodEnter
    private static void before(@Advice.AllArguments Object[] args,  @Advice.Origin("#m") String methodName, @Advice.This Object thiz) {          
        FilterHolder filterHolder = ((org.eclipse.jetty.servlet.ServletHandler)thiz).addFilterWithMapping(XYZFilter.class, "/*", EnumSet.of(javax.servlet.DispatcherType.REQUEST));
    }

Answer 1

Byte Buddy is a code generation framework and is not aspect-oriented. Think of the code being copy-pasted to the target location; the stack overflow error you are seing would be the same if you hardcoded the instrumentation into your target method.

This can be avoided by adding a flag, for example, you could define a ThreadLocal<Boolean> that you set to true before making a recursive call, for example:

if (!threadLocal.get()) {
  threadLocal.set(true);
  try {
    // your code here.
  } finally {
    threadLocal.set(false);
  }
}

This way, you can keep track of a recursive call. You do however need to manage your state somehome. One option would be to inject a holder class for your property into the bootstrap class loader using the Instrumentation interface.

Alternatively, you can check the stack for a recurive call. This is not as efficient as the explicit state management but starting with Java 9, you can use the stack walker API which is much cheaper and makes this approachable.