I have a custom annotation to filter out tests at run-time, based on the characteristics of the device-under-test. The annotation can be applied to test classes and to test methods.
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface PhysicalKeyboardTest {
boolean keyboardRequired() default false;
}
To filter out the annotated tests I have a custom test runner:
public class MyTestRunner extends BlockJUnit4ClassRunner {
public MyTestRunner(Class<?> klass) throws InitializationError {
super(klass);
}
@Override
protected List<FrameworkMethod> computeTestMethods() {
return filterKeyboardRequiredTests(super.computeTestMethods());
}
private List<FrameworkMethod> filterKeyboardRequiredTests(List<FrameworkMethod> allTests) {
// create a List that we can modify
List<FrameworkMethod> filteredTests = new ArrayList<>(allTests);
// does the test class require a keyboard?
if (isKeyboardRequired(getTestClass())) {
// test class is marked "keyboardRequired", filter out all tests
// PROBLEM: this code causes test-time 'initializationError'
filteredTests.clear();
return filteredTests;
}
// for each test: does it require a keyboard?
for (Iterator<FrameworkMethod> iterator = filteredTests.iterator(); iterator.hasNext(); ) {
FrameworkMethod test = iterator.next();
// does the test require a keyboard?
if (isKeyboardRequired(test)) {
// test is marked "keyboardRequired", filter it out
iterator.remove();
}
}
return filteredTests;
}
/**
* Determine if the given test class or test is annotated with {@code keyboardRequired}
*
* @param annotatable The test class or test
* @return True if so annotated
*/
private boolean isKeyboardRequired(Annotatable annotatable) {
PhysicalKeyboardTest annotation = annotatable.getAnnotation(PhysicalKeyboardTest.class);
return annotation != null && annotation.keyboardRequired();
}
The code works as expected for individual test methods that are annotated.
However if the test class is annotated, when the tests are run I get an initializationError
java.lang.Exception: No runnable methods
at org.junit.runners.BlockJUnit4ClassRunner.validateInstanceMethods(BlockJUnit4ClassRunner.java:191)
at org.junit.runners.BlockJUnit4ClassRunner.collectInitializationErrors(BlockJUnit4ClassRunner.java:128)
at org.junit.runners.ParentRunner.validate(ParentRunner.java:416)
at org.junit.runners.ParentRunner.<init>(ParentRunner.java:84)
at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:65)
at com.winterberrysoftware.luthierlab.testFramework.MyTestRunner.<init>(MyTestRunner.java:32)
at java.lang.reflect.Constructor.newInstance(Native Method)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at androidx.test.internal.runner.junit4.AndroidAnnotatedBuilder.runnerForClass(AndroidAnnotatedBuilder.java:63)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at androidx.test.internal.runner.AndroidRunnerBuilder.runnerForClass(AndroidRunnerBuilder.java:153)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:73)
at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:793)
at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:547)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:390)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)
This error is probably due to computeTestMethods() returning an empty list. (It fails even more dramatically if a null value is returned.)
It seems likely that filtering on the class-level annotation should be done elsewhere (probably where the list of test classes is created), but I have not been able to find where to do it.
Thanks for any help.
I figured it out.
Instead of providing a custom BlockJUnit4ClassRunner, I need to provide a custom AndroidJUnitRunner, in conjunction with a custom filter (associated with the annotation).
I added an inner class KeyboardFilter to my custom annotation. The custom annotation now looks like:
/**
* This annotation is used to mark tests for devices with a physical
* keyboard (Chromebook).
* <br><br>
* The {@code keyboardRequired} param can be used to flag tests that
* should not be run if a physical keyboard is not present.
* <br><br>
* The annotation can be applied to test classes, and to individual
* tests.
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface PhysicalKeyboardTest {
boolean keyboardRequired() default false;
/**
* When {@code keyboardRequired=true} and the test is running on a
* device that does not have a physical keyboard, filter the test out.
* <br><br>
* Will be instantiated via reflection by
* {@link RunnerArgs.Builder#fromBundle(android.app.Instrumentation, android.os.Bundle)}
*/
@SuppressWarnings("unused")
class KeyboardFilter extends ParentFilter {
/**
* Determine if the given test can run on the current device, based
* on the presence of a physical keyboard.
* <br><br>
* Tests that are annotated with {@code @PhysicalKeyBoardTest(keyboardRequired=true)}
* can only be run on devices with a physical keyboard (i.e. Chromebook).
*
* @param description the {@link Description} describing the test
* @return <code>true</code> if test can run
*/
@Override
protected boolean evaluateTest(Description description) {
if (TestHelpers.isChromebook()) {
return true;
}
// we are not running on a Chromebook (i.e. no physical keyboard is attached)
// check for test-class and test-method annotations
PhysicalKeyboardTest testAnnotation = description.getAnnotation(PhysicalKeyboardTest.class);
PhysicalKeyboardTest classAnnotation = description.getTestClass().getAnnotation(PhysicalKeyboardTest.class);
// if the test-method and test-class are not annotated, the test can run
return noKeyboardRequired(testAnnotation) && noKeyboardRequired(classAnnotation);
}
@Override
public String describe() {
return "skip tests annotated with 'PhysicalKeyboardTest(keyboardRequired=true)' " +
"if no physical keyboard is present";
}
/**
* Determine if the given annotation says that a keyboard is required.
*
* @param annotation The annotation
* @return True if no keyboard is required
*/
private boolean noKeyboardRequired(PhysicalKeyboardTest annotation) {
return annotation == null || !annotation.keyboardRequired();
}
}
}
/**
* An instrumentation test runner.
* <br><br>
* Provides a mechanism for filtering out test classes and test methods,
* based on a custom test annotation.
* <br><br>
* This class is specified as the {@code testInstrumentationRunner}
* in the app's {@code build.gradle} file.
*
* @see PhysicalKeyboardTest
* @see androidx.test.internal.runner.RunnerArgs
*/
@SuppressWarnings("unused")
public class MyAndroidJUnitRunner extends AndroidJUnitRunner {
// androidx.test.internal.runner.RunnerArgs looks for this bundle key
private static final String FILTER_BUNDLE_KEY = "filter";
@Override
public void onCreate(final Bundle bundle) {
// add the keyboard filter to the test runner's filter list
bundle.putString(FILTER_BUNDLE_KEY, PhysicalKeyboardTest.KeyboardFilter.class.getName());
super.onCreate(bundle);
}
}
In the app's build.gradle I specify my instrumentation test runner:
testInstrumentationRunner "com.mypath.testFramework.MyAndroidJUnitRunner"
With this framework in place, my tests that include the annotation
@PhysicalKeyboardTest(keyboardRequired = true)
will not be run on devices that do not have a keyboard. The annotation can be applied to the test class, or to individual test methods.