Search code examples
javatestingjenkinstestngsurefire

Jenkins mixing up tests from different Java classes


We have a project with lots of tests running against a database. Many of the tests set up the database using @BeforeClass, and clean it out using @AfterClass, which works fine when we run the tests locally. When Jenkins runs them, it mixes up the order so tests from different classes are intermixed. For instance, A.TestAlpha, B.TestFoo, A.TestGamma. Then inevitably the tests fail because they don't have the correct setup. My understanding of Jenkins is that it's supposed to run one class of tests at a time. Is there some config somewhere that might tell it to run the tests in this weird way?

(edit) The same sort of problems occur running tests locally with mvn. From mvn help:effective-pom

(project)
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.17</version>
      <configuration>
        <includes>
          <include>**/When*.java</include>
          <include>**/Test*.java</include>
          <include>**/*Test.java</include>
          <include>**/*TestCase.java</include>
        </includes>
      </configuration>
    </plugin>
(module)
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.17</version>
      <executions>
        <execution>
          <id>default-test</id>
          <phase>test</phase>
          <goals>
            <goal>test</goal>
          </goals>
          <configuration>
            <includes>
              <include>**/When*.java</include>
              <include>**/Test*.java</include>
              <include>**/*Test.java</include>
              <include>**/*TestCase.java</include>
            </includes>
          </configuration>
        </execution>
      </executions>
      <configuration>
        <includes>
          <include>**/When*.java</include>
          <include>**/Test*.java</include>
          <include>**/*Test.java</include>
          <include>**/*TestCase.java</include>
        </includes>
      </configuration>
    </plugin>

Solution

  • Thanks to everyone who attempted to help. The bug apparently lies in TestNG.

    How to make sure TestNG runs methods on test classes in succession instead of interleaved?