Best practice with codeception fixture performance
At this moment my codeception tests execute half of hour, and obviously this time grows when I add new tests.
I try to achieve much better performance with my tests because after push I very often simply forget that I pushed something because of slow tests.
At this moment I see that my biggest problem is fixtures. Here very descriptive blackfire profile:
Only loading and unloading fixtures take 93% of the time, but the testing itself takes 5% of the time.
I thought that I could use a single set of fixtures for all my test and just rollback transactions after every test, but I found that with a large number of test cases it is becoming more and more difficult to maintain such data - adding new tests cases always broke some old tests.
So, any solution how can I decrease the time consumed by fixtures? Maybe some way here to use raw SQL dump for each set of tests? Or should I write a huge shell script that restore dumps to the database and run each set of tests separately?
There is no simple answer to this. Here are a few things that worked for me in the past. Some might not work with your application or maybe doesn't make sense. I hope that here is something useful for you.
Reduce the number of fixtures per test you have to load
Define a set of common data (users, products, whatever you have in your application) that can be used by all tests. It doesn't have to be a "one for all" set. Think more of it as the common denominator. This data should not be modified in your tests or at least not in a way that breaks tests. Load this fixture before running the test suite. In each test, load only the additional fixtures you need for this specific test.
Replace Yii Fixtures
I'm not familiar with Yii, but framework fixture loaders are usually not that efficient. Maybe you would be better off with some own tooling.
Switch to an in-memory database
It's usually a bit to a whole lot faster.
Copy the database
You can reduce the loading time by loading the fixture once, creating a copy of the database and then just reloading it. SQL dumps are one way to achieve this. With SQLite, you can simply create a backup of the database file and copy it back before each test.
If I got you right, you have sets of tests that share the same data. You could create a suite for each set. Add a test listener that loads the fixture and creates the copy before the suite runs and restore the backup before each test.
Unit test the use cases
Not exactly what you asked for, but this will drastically reduce the time needed to run your tests. When you have all your use cases unit tested, you'll need much fewer end-to-end tests. Depending on your architecture this may not be feasible at the moment, but it's never too late to start working towards it.