Why does JUnitCore only return the first results?

Question

I am using JUnit 4 with Eclipse JDT to create automated Mutant testing.

Here is the general overview of my code structure:

//Basic for loop
for(int i = 0; i < 10; i++) {

    //Read in source code from a Java file
    ...(works)

    //Change a line using JDT and save code to a new Java file
    ...(works)

    //Compile new Java file (this also works)
    try {       
        Process compile = Runtime.getRuntime().exec("javac -cp \"src/*\" " + path + "*.java");
        compile.waitFor();
    } catch(IOException ex) { /*...*/ }
      catch(InterruptedException ex) { /*...*/ }

    //Run JUnit Tests (this works the first time it is called)
    JUnitCore core = new JUnitCore();
    Result result = core.run(JUnitTest.class); //This class contains my JUnit Tests
}

My code above works for the first test, but every test after that always returns the same results. Why are new results not generated even though different mutations are made?

Things I have tried:

1. Testing that different mutations are made at every loop iteration.

2. Testing that the new code is compiled before the test is run.

3. Running the internals of the for loop as a thread, wait for that thread to finish, then run the next test.

4. Using JUnitCore.runClasses(JUnitTest.class) in lieu of creating an instance of core and calling core.run(JUnitTest.class):

   JUnitCore core = new JUnitCore();
   Result result = core.run(JUnitTest.class);
   

5. Substituting JUnitCore (org.junit) code for TestRunner (junit.textui), which gave me the same problem:

   TestSuite suite= new TestSuite();
   suite.addTestSuite(JUnitTest.class);
   TestResult result = TestRunner.run(suite);
   

Answer 1

You need to insert the mutant into the JVM - although you are compiling the modified file the JVM will only see the first loaded version.

There are various ways to do this, from launching a new JVM for each mutant through to using the instrumentation API.