Why does my Java JUnit 5 Timeout Test PASS when the execution in milliseconds exceeds what was specified?

Question

What i'm trying to do:

Fail the test after 2 milliseconds

Here's an example of my code. It varies between 12-22 ms. I've discovered that if i change the value to '1' it will sometimes even fail at seemingly random times..

@Test
@Timeout(unit = TimeUnit.MILLISECONDS, value = 2)

public void hundredCities() {

    assertEquals(740,
            stringAlgorithm.findTotalLengthOfStringArray(
                    new String[] {
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam" })
    );

}

However...

Test passed

What am i doing wrong here?

String algorithm code:

public class StringAlgorithm {

public int findTotalLengthOfStringArray(String[] test) {
    // String[] test = {"Bananas", "Apples"};
    String s = "";

    for (String i : test) {
        s += i;
    }

    char [] c = s.toCharArray();
    System.out.println(c.length);

    return c.length;

}

}

Answer 1

The time reported by IntelliJ IDEA may not accurately reflect the execution time of the test method. In particular, the time reported for the first test executed includes time spent setting up the test suite. You can see by adding another test method earlier in the test class that the time "moves" to that test, and hundredCities itself may very well execute within the 2ms timeout duration.

You can verify this by logging and asserting the time taken explicitly:

@Test
public void hundredCities() {
    long start = System.nanoTime();
    assertEquals(740,
            stringAlgorithm.findTotalLengthOfStringArray(
                    new String[] {
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam",
                            "Stockholm", "Paris", "London", "New York", "Amsterdam" })
    );
    long executionTime = System.nanoTime() - start;
    System.out.println(Duration.ofNanos(executionTime).toMillis());
    assertTrue(executionTime < 2_000_000);
}

On my machine, I had to add around a thousand cities before it consistently took greater than 2ms. Of course, that threshold will vary on different hardware, operating systems, Java versions, etc.