Test failing when run with XCTool with logicTestBucketSize

Question

I have a project with over 1000 unit test and was thinking to speed up the build by using the xctool's parallelise option.

So i turned that on and set logicTestBucketSize to 50. The test run, but some are failing which are not failing when not using this option.

My question: are buckets run independently in their own sandbox or do they share global variable that the unit test might set up? which might explain some cross contamination between the tests

Answer 1

Yes. When running tests in parallel, xctool will run each bucket of tests in a single process, and run multiple buckets simultaneously in different processes. Additionally, you can select whether bucketing will be done on a case or class basis with -bucketBy class. You should probably use class unless you have very large test classes with many test cases.

Your tests may fail now though it didn't before because:

1. A test case relies on global state set up by a previous test case, even from a different test class, as long as it is grouped into the same binary. This test would now fail as the order the tests run in may be different, or not run at all.
2. A test alters global state and cause later tests to fail. This may not have been a problem before because that test was run after other tests that might be affected have already ran.

A good way of dealing with the first type of failures is run with bucket size 1 (either bucket-by-class mode or bucket-by-case mode, depending on what mode you'll be running later).