Randomise the order of instrumentation tests

Question

I'm wondering if it's possible to randomise the order in which instrumentation tests are run, i.e. those extending ActivityInstrumentationTestCase2. I tried following this blog post, but I can't work out how to tell the testing framework that I wish to use my test runner.

The problem is that I can't use the @RunWith annotation, as these are (as I understand it) JUnit3 tests, rather than JUnit4.

It's quite possible that this is pointless, as they don't need to be randomised, but it would be nice to prove the tests' independence in this way.

Ideally I'd like to get it running first using the command line and the gradle wrapper.

Then, it would be nice to have it working via Android Studio, if possible.

[Edit]

I can see that when you do "Edit Configurations . . ." in AS, it's possible to specify your own runner there, via the "Specific instrumentation runner (optional)" box. Unfortunately if I do that, I get the following error:

Test running startedTest running failed: Unable to find instrumentation info for: ComponentInfo{<path_to_class_here>.RandomizingTestRunner}
Empty test suite.

And I can't work out why.

Answer 1

You could use the following randomized runner:

package com.example.test.runners;

import android.test.InstrumentationTestRunner;
import android.test.suitebuilder.TestSuiteBuilder;
import junit.framework.Test;
import junit.framework.TestSuite;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class RandomizedInstrumentationTestRunner extends InstrumentationTestRunner {

 @Override
 public TestSuite getTestSuite() {
    return buildTestSuite();
 }

 private TestSuite buildTestSuite() {
    TestSuiteBuilder builder = new TestSuiteBuilder(getClass().getName(), getTargetContext().getClassLoader());
    builder.includePackages("");

    List<Test> tests = new ArrayList<Test>();
    addTestsFromSuite(builder.build(), tests);
    Collections.shuffle(tests);

    TestSuite randomizedSuite = new TestSuite();
    for (Test one : tests) {
        randomizedSuite.addTest(one);
    }

    return randomizedSuite;
 }


 private void addTestsFromSuite(TestSuite suite, List<Test> out) {
    List<Test> tests = Collections.list(suite.tests());
    for (Test one : tests) {
        if (one instanceof TestSuite) {
                 addTestsFromSuite((TestSuite) one, out);
             }
             else{
                 out.add(one);
             }
        }
  }
}

and don't forget to set the runner in your build.gradle file:

android {

   defaultConfig {
       testInstrumentationRunner "com.example.test.runners.RandomizedInstrumentationTestRunner"
       minSdkVersion 8
   }

   ....
}

Finally run the following twice to verify the random order of execution:

./gradlew connectedCheck --info