Make Julia testsuite report all error messages at end of run

Question

I'm playing around with the Julia testsuite functionality and I quite like it. What i however fail to figure out is how to make the testsuite report the potential message upon completion.

Let's say I have a test suite like the one below, where each of the functions performs a bunch of tests in turn before returning

@testset "MyTestSuite" begin
    @testset "Subtest1" begin @test my_test_1() end
    @testset "Subtest2" begin @test my_test_2() end
    @testset "Subtest3" begin @test my_test_3() end
    @testset "Subtest4" begin @test my_test_4() end
end

If now say my_test_4 fails or throws an error, then the ouput would look like below

Test Summary:                                   |    Pass  Error    Total
MyTestSuite                                     |      65      1       66
  Subtest1                                      |       5      1        6
  Subtest2                                      |      10      0       10
  Subtest3                                      |      20      0       20
  Subtest4                                      |      30      0       30
ERROR: LoadError: Some tests did not pass: 65 passed, 0 failed, 1 errored, 0 broken.

But there is now way (to my knowledge) for me to see what went wrong without scrolling upwards in the terminal output. If my test suites are long enough, and produce enough diagnostics, the error information could even be lost to me or at least very difficult to find.

So, does anyone know a nice way around this? Are there options that one can give to the macro @testset to make sure it prints of collects the errors for further processing?

Answer 1

You can define your custom AbstractTestSet. It is descirbed here in the Julia manual.

Here is an example adapted from the manual. First define:

using Test

struct CustomTestSet <: Test.AbstractTestSet
    description::AbstractString
    results::Vector
    CustomTestSet(desc) = new(desc, [])
end

Test.record(ts::CustomTestSet, child::Test.AbstractTestSet) = push!(ts.results, child)
Test.record(ts::CustomTestSet, res::Test.Result) = push!(ts.results, res)

function Test.finish(ts::CustomTestSet)
    if Test.get_testset_depth() > 0
        Test.record(Test.get_testset(), ts)
    end
    ts
end

and now you can write:

julia> res = @testset CustomTestSet "custom testset" begin
           # this testset should inherit the type, but not the argument.
           @testset "custom testset inner" begin
               @test 1==1
               @test 1==2
               @test 2==2
               @test 2==3
           end
       end
CustomTestSet("custom testset", Any[CustomTestSet("custom testset inner", Any[Test Passed, Test Failed at REPL[10]:5
  Expression: 1 == 2
   Evaluated: 1 == 2, Test Passed, Test Failed at REPL[10]:7
  Expression: 2 == 3
   Evaluated: 2 == 3])])

julia> res.results[1].results
4-element Array{Any,1}:
 Test Passed
 Test Failed at REPL[10]:5
  Expression: 1 == 2
   Evaluated: 1 == 2
 Test Passed
 Test Failed at REPL[10]:7
  Expression: 2 == 3
   Evaluated: 2 == 3

and you have an access to a vector that tells you what passed and what failed, and on failure what was the problem.

You can also filter out passed tests:

julia> filter(x -> !(x isa Test.Pass), res.results[1].results)
2-element Array{Any,1}:
 Test Failed at REPL[6]:5
  Expression: 1 == 2
   Evaluated: 1 == 2
 Test Failed at REPL[6]:7
  Expression: 2 == 3
   Evaluated: 2 == 3

If your tests have a more complex nested structure you should do it recursively.

Is this something you wanted?