I'm working with a lot of legacy code. There was a JUnit-TestSuite to begin with. When running all tests with gradle, they failed. When running the test in IntelliJ, they worked. We configured gradle to use the test suite.
Now someone reported tests working locally without gradle, but not with gradle. It's time we fix this mess.
Is there a smart way to figure out which test leaves some configuration behind or which tests relies on the other tests?
The most likely cause of this "bleed" from one test into another is mutable static values. By default, all tests are run by the same JVM so a static variable which is "mutated" by one test will be "dirty" in another test.
Mutable statics are evil! I'm working on a codebase currently with mutable statics everywhere and it's a mess. If possible you should refactor to use dependency injection and store mutable state in instances and not statics.
The best workaround is to find the tests which "dirty" the static mutable variables and do
@After
public void cleanup() {
SomeStatic.reset();
}
If you can't find the "dirty" test which is causing the issue, you might be forced to do the following in the "failing" test. This is not preferred, and a little hacky
@Before
public void cleanBefore() {
SomeStatic.reset();
}
But this has a slight code "smell". Better to find the offending test which "dirties" the mutable static
The "nuclear" option is to run each test in its own jvm. This is a total hack and should be avoided at all costs. It will drastically increase the time it takes to run your tests
test {
forkEvery = 1
}
See Test.forkEvery