Run benchmark on method using completable future

Question

I'm trying to measure the performance of a particular method. I run the benchmarks just fine when calling the method directly, but when the method used a completable future with a custom executor everything collapsed. I've implemented the method to use a completable future in order to force a timeout if the method takes too long.

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Threads(value = 5)
@Warmup(iterations = 20)
@Measurement(iterations = 50, timeUnit = TimeUnit.MILLISECONDS)
public String very_big_query(TestState testState) throws Exception {
    return testState.transpiler.process(testState.veryBigQuery);
}

---

@State(Scope.Thread)
public static class TestState {
    String veryBigQuery;
    Transpiler transpiler;

    @Setup(Level.Trial)
    public void doSetupTrial() throws Exception {
        veryBigQuery = "(";
        for(int i = 0; i < 99; i++) {
            veryBigQuery += String.format("java_%s OR ", i);
        }
        veryBigQuery += "java_100) AND (";
        for(int i = 100; i < 199; i++) {
            veryBigQuery += String.format("java_%s OR ", i);
        }
        veryBigQuery += String.format("java_%s)", 200);
    }

    @Setup(Level.Invocation)
    public void doSetupInvocation() throws Exception {
        random = ThreadLocalRandom.current().nextInt(0, productionQueries.size());
        randomQuery = productionQueries.get(random);
        transpiler = new Transpiler(5, 100); //number of threads in custom pool for the executor, timeout in milliseconds
    }
}

---

public String process(final String input) throws Exception {
    CompletableFuture<String> cf = CompletableFuture.supplyAsync(() -> {
        return SOME_STRING;
    }, executor);

    return cf.get(timeoutInMillis, TimeUnit.MILLISECONDS);
}

---

this.executor = Executors.newFixedThreadPool(numberOfThreads);

I get this error

JMH had finished, but forked VM did not exit, are there stray running threads? Waiting 24 seconds more...

Non-finished threads:

Can someone explain to me why is this happening and how do I approach it in order to make it work?

Answer 1

First be sure to go through the samples of JMH, it is very easy to shoot yourself in the foot with these configs (I write tests via JMH once a week probably) and still mess things up every time.

Then, get rid of Level.Invocation unless you really understand what it does...

And last may be shut down the executor in a @TearDown method.