I am trying to write own moduleUnitTester which will execute all tests in module even if one fails. default unit tester works like this:
size_t failed = 0;
foreach (m; ModuleInfo) {
if (m) {
auto fp = m.unitTest;
if (fp) {
try {
fp();
}
catch (Throwable e) {
writeln(e);
failed++;
}
}
}
}
the fp() throws on first failure. I dont really like that, the m.unitTest returns void function which is a function that will execute all unit tests in a module. Is there any way to list these unit tests and iterate over each one? this does not work:
foreach (m; ModuleInfo)
{
__traits(getUnitTests, m);
}
That would let me grab all the unit tests and then iterate over them freely. Says the 'm' is a variable not a module. I could not find any documentation what is 'ModuleInfo' actually I found this only by mistake...
Well, since I wrote half the answer in the comments anyway, I guess I might as well write a bit more in here too.
There's a few options here: compile time and run time. The compile time ones give you access to UDAs and other broken down details per module, but don't do a good job giving you access to all the modules. You can try to walk the import graph, but local imports are not available to this method. You can use a build tool to list your modules, but this, of course, requires you to actually use a build tool with that feature (https://github.com/atilaneves/unit-threaded is a lib that does this with dub).
You can also pass a manual list of modules to your test runner, which is the most maintenance work, but might be nice flexibility.
But, I want to do it at runtime, like you have in the question, just with more detail. How? By doing some low-level pointer stuff! It sometimes still pays to be an old assembly language hacker :)
Behold the code with inline comments:
module q.test;
unittest {
assert(0, "Test 1 failed");
}
unittest {
assert(true);
}
// module ctor to set our custom test runner instead of the default
shared static this() {
import core.runtime;
Runtime.moduleUnitTester = &myhack;
}
bool myhack() {
/*
OK, so here's the situation. The compiler will take each unittest block
and turn it into a function, then generate a function that calls each
of these functions in turn.
core.runtime does not give us access to the individual blocks... but DOES
give us the unitTest property on each module compiled in (so we catch them
all automatically, even with separate compilation, unlike with the CT
reflection cases) which is a pointer to the auto-generated function-calling
function.
The machine code for this looks something like this:
0000000000000000 <_D1q4test9__modtestFZv>:
0: 55 push rbp
1: 48 8b ec mov rbp,rsp
4: e8 00 00 00 00 call 9 <_D1q4test9__modtestFZv+0x9>
9: e8 00 00 00 00 call e <_D1q4test9__modtestFZv+0xe>
e: 5d pop rbp
f: c3 ret
The push and mov are setting up a stack frame, irrelevant here. It is the
calls we want: they give us pointers to the individual functions. Let's dive in.
*/
bool overallSuccess = true;
foreach(mod; ModuleInfo) {
// ModuleInfo is a runtime object that gives info about each
// module. One of those is the unitTest property, a pointer
// to the function described above.
if(mod.unitTest) {
// we don't want a function, we want raw bytes!
// time to cast to void* and start machine code
// hacking.
void* code = mod.unitTest();
version(X86_64) {
code += 4; // skip function prolog, that push/mov stuff.
} else version(X86) {
code += 3; // a bit shorter on 32 bit
} else static assert(0);
// Opcode 0xe8 is the 32-bit relative call,
// as long as we see those calls, keep working.
while(* cast(ubyte*) code == 0xe8) {
code++; // skip the opcode...
// ...which lands us on the relative offset, a 32 bit value
// (yes, it is 32 bit even on a 64 bit build.)
auto relative = *(cast(int*) code);
// the actual address is the next instruction add + the value,
// so code+4 is address of next instruction, then + relative gets
// us the actual function address.
void* address = (code + 4) + relative;
auto func = cast(void function()) address;
// and run it, in a try/catch so we can handle failures.
try {
func();
import std.stdio;
writeln("**Test Block Success**");
} catch(Throwable t) {
import std.stdio;
writeln("**Failure: ", t.file, ":", t.line, " ", t.msg);
overallSuccess = false;
}
// move to the next instruction
code += 4;
}
}
}
// returning false means main is never run. When doing a
// unit test build, a lot of us feel running main is just
// silly regardless of test passing, so I will always return
// false.
// You might want to do something like C exit(1) on failure instead
// so a script can detect that.
return false && overallSuccess;
}
I'll leave pulling debug symbols out to print file+line info of succeeding test as an exercise for the reader, if desired.
I provide this filthy hack with the hope that it will be useful, but WITHOUT WARRANTY of any kind, not even merchantability or fitness for a particular purpose.
I tested with dmd on Linux, it might or might not work elsewhere, I don't know if gdc and ldc generate the same function, or if their optimizations make a difference, etc.
I'd recommend using supported techniques like the build tool or manually maintained module list combined with compile time reflection if you really want a new test runner: that unit-threaded library does a lot of nifty things beyond this too, so do check it out.
But the runtime-only option isn't exactly a dead end either :)