ceylon.test.TestRunner fails when tests fails

Question

Whenever a test function (a function annotated with test) contains assertions that fail, the assertion has the same effect as when trowing an exception: no further code lines in that function will be executed. Thus, assert statements in functions that are annotated with 'test' works just as ordinary assert statements in ordinary Ceylon functions. This runs contrary to the documentation, which states that ordinary assert statements can be used for making unit tests.

Thus, running the code below, I get to see the statement myTests1 but not ´myTests2`:

import ceylon.test {
    test, TestRunner, createTestRunner
}

test
Anything myTests1 () {
    // assert something true!
    assert(40 + 2 == 42);
    print("myTests1");
    return null;
}

test
void myTests2 () {
    // assert something false!
    assert(2 + 2 == 54);
    print("myTests2");
}


"Run the module `tests`."
shared void run() {

    print("reached run function");

    TestRunner myTestRunner = createTestRunner(
        [`function myTests1`, `function myTests2`]);

    myTestRunner.run();
}

This is the actual output:

"C:\Program Files\Java\jdk1.8.0_121\bin\java" -Dceylon.system.repo=C:\Users\Jon\.IdeaIC2017.2\config\plugins\CeylonIDEA\classes\embeddedDist\repo "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2017.2.1\lib\idea_rt.jar=56393:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2017.2.1\bin" -Dfile.encoding=windows-1252 -classpath C:\Users\Jon\.IdeaIC2017.2\config\plugins\CeylonIDEA\classes\embeddedDist\lib\ceylon-bootstrap.jar com.redhat.ceylon.launcher.Bootstrap run --run run tests/1.0.0
reached run function
myTests1

Process finished with exit code 0

Answer 1

This is working as intended – replacing those asserts with assertEquals’ has the same effect and prints the same output, because both do exactly the same thing: throw an exception if the assertion fails.

All test frameworks that I’m aware of behave the same way in this situation: an assertion failure results in an exception and thus immediately terminates execution of the test method. This is by design, since you don’t know what your program will do once one expectation will be violated – the rest of the method might depend on that assertion holding true, and might break in unpredictable and confusing ways.

If you’re writing tests like

test
shared void testTwoThings() {
    assertEquals { expected = 42; actual = 40 + 2; };
    assertEquals { expected = 42; actual = 6 * 9; };
}

you’re supposed to write two tests instead.