I'm trying to log my test suite results using AspectJ. I'd like to 'inject' the results identification code after each @Test method in my code, so created an aspect with the following method:
@After("execution(* *(..)) && @annotation(org.junit.Test)")
public void afterTestMethod(JoinPoint joinPoint) {
//identify test result
}
However, can't find how to retrieve the test method result (passed/failed/skipped). Any suggestions?
Thanks!
I am assuming that you build your project with something like Maven or Gradle and will show you a Maven example for a JUnit 4 RunListener:
In module test-tools you would add
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<!--
We build something with the JUnit API, not just run a test,
so scope 'test' is not enough here
-->
<scope>compile</scope>
</dependency>
to your POM and then have something like this in src/main/java/...:
package de.scrum_master.testing;
import org.junit.runner.Description;
import org.junit.runner.Result;
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class ResultPrintingRunListener extends RunListener {
@Override
public void testRunStarted(Description description) {
System.out.println("[RunStarted] description = " + description);
}
@Override
public void testRunFinished(Result result) {
System.out.println("[RunFinished] result = " + result);
System.out.println(" run count = " + result.getRunCount());
System.out.println(" failure count = " + result.getFailureCount());
System.out.println(" assumption failure count = " + result.getAssumptionFailureCount());
System.out.println(" ignored count = " + result.getIgnoreCount());
System.out.println(" run time (ms) = " + result.getRunTime());
}
@Override
public void testSuiteStarted(Description description) {
System.out.println("[SuiteStarted] description = " + description);
}
@Override
public void testSuiteFinished(Description description) {
System.out.println("[SuiteFinished] description = " + description);
}
@Override
public void testStarted(Description description) {
System.out.println("[Started] description = " + description);
}
@Override
public void testFinished(Description description) {
System.out.println("[Finished] description = " + description);
}
@Override
public void testFailure(Failure failure) {
System.out.println("[Failure] failure = " + failure);
}
@Override
public void testAssumptionFailure(Failure failure) {
System.out.println("[AssumptionFailure] failure = " + failure);
}
@Override
public void testIgnored(Description description) {
System.out.println("[Ignored] description = " + description);
}
}
Instead of logging you would put your API calls in there at the appropriate places.
In your application test module you would add the test-tools module as a dependency and then configure your Maven Surefire/Failsafe plugins like this:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<configuration>
<properties>
<property>
<name>listener</name>
<value>de.scrum_master.testing.ResultPrintingRunListener</value>
</property>
</properties>
</configuration>
</plugin>
If then you run a test like this in Maven, ...
package de.scrum_master.agent.aspect;
import org.junit.Assert;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
public class MyTest {
@Test
public void one() {
Assert.assertEquals("xander", "Alexander".substring(3));
}
@Test
public void two() {
Assert.assertEquals("Alex", "Alexander".substring(3));
}
@Test
public void three() {
Assert.assertEquals(11, 1 / 0);
}
@Test
@Ignore
public void four() {
Assert.assertNull(null);
}
}
... Maven prints:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[RunStarted] description = null
[INFO] Running de.scrum_master.agent.aspect.MyTest
[SuiteStarted] description = de.scrum_master.agent.aspect.MyTest
[Started] description = one(de.scrum_master.agent.aspect.MyTest)
[Finished] description = one(de.scrum_master.agent.aspect.MyTest)
[Started] description = two(de.scrum_master.agent.aspect.MyTest)
[Failure] failure = two(de.scrum_master.agent.aspect.MyTest): expected:<[Alex]> but was:<[xander]>
[Finished] description = two(de.scrum_master.agent.aspect.MyTest)
[Ignored] description = four(de.scrum_master.agent.aspect.MyTest)
[Started] description = three(de.scrum_master.agent.aspect.MyTest)
[Failure] failure = three(de.scrum_master.agent.aspect.MyTest): / by zero
[Finished] description = three(de.scrum_master.agent.aspect.MyTest)
[SuiteFinished] description = de.scrum_master.agent.aspect.MyTest
[ERROR] Tests run: 4, Failures: 1, Errors: 1, Skipped: 1, Time elapsed: 0.11 s <<< FAILURE! - in de.scrum_master.agent.aspect.MyTest
[ERROR] de.scrum_master.agent.aspect.MyTest.two Time elapsed: 0.007 s <<< FAILURE!
org.junit.ComparisonFailure: expected:<[Alex]> but was:<[xander]>
at de.scrum_master.agent.aspect.MyTest.two(MyTest.java:31)
[ERROR] de.scrum_master.agent.aspect.MyTest.three Time elapsed: 0.001 s <<< ERROR!
java.lang.ArithmeticException: / by zero
at de.scrum_master.agent.aspect.MyTest.three(MyTest.java:36)
[RunFinished] result = org.junit.runner.Result@79be0360
run count = 3
failure count = 2
assumption failure count = 0
ignored count = 1
run time (ms) = 0
If you prefer something which is independent of a specific JUnit runner right inside your tests, create a base class like this using JUnit TestWatcher:
package de.scrum_master.agent.aspect;
import org.junit.Rule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
public class TestBase {
@Rule(order = Integer.MIN_VALUE)
public TestWatcher testWatcher = new TestWatcher() {
@Override
protected void failed(Throwable e, Description description) {
System.out.println("[TestWatcher failed] description = " + description +", e = " + e);
}
@Override
protected void succeeded(Description description) {
System.out.println("[TestWatcher succeeded] description = " + description);
}
};
}
Then make all your test classes extends TestBase directly or indirectly. If you run the test, e.g. from an IDE, you see (shortened output, only test watcher log):
[TestWatcher succeeded] description = one(de.scrum_master.agent.aspect.MyTest)
[TestWatcher failed] description = two(de.scrum_master.agent.aspect.MyTest), e = org.junit.ComparisonFailure: expected:<[Alex]> but was:<[xander]>
[TestWatcher failed] description = three(de.scrum_master.agent.aspect.MyTest), e = java.lang.ArithmeticException: / by zero
You see, there are fewer events in the test watcher, e.g. ignored tests are not reported.
As much as I love AspectJ (which is how I found this question), I think you should try configuring JUnit appropriately first and will probably be happy with it. If for whatever reason - please explain, if so - you still insist in an AOP solution, please let me know.
Because you insisted:
package de.scrum_master.aspectj;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.AfterReturning;
import org.aspectj.lang.annotation.AfterThrowing;
import org.aspectj.lang.annotation.Aspect;
@Aspect
public class TestResultInterceptor {
@AfterReturning(value = "execution(* *(..)) && @annotation(org.junit.Test)", returning = "result")
public void allMethods(JoinPoint joinPoint, Object result) {
System.out.println(joinPoint + " -> PASSED");
}
@AfterThrowing(value = "execution(* *(..)) && @annotation(org.junit.Test)", throwing = "throwable")
public void allMethods(JoinPoint joinPoint, Throwable throwable) {
System.out.println(joinPoint + " -> FAILED: " + throwable);
}
}
When running the JUnit test above in my IDE, the console log is:
execution(void de.scrum_master.testing.MyTest.one()) -> PASSED
execution(void de.scrum_master.testing.MyTest.two()) -> FAILED: org.junit.ComparisonFailure: expected:<[Alex]> but was:<[xander]>
execution(void de.scrum_master.testing.MyTest.three()) -> FAILED: java.lang.ArithmeticException: / by zero
I think you can take it from here.