My one test file with 7 empty tests is taking 5+ seconds to start running. I've used both flutter test and the Dart test runner in VS Code, and they both take about the same amount of time.
My whole test suite of ~150 tests takes over 30 seconds to complete, because each test file is taking 3-7 seconds just to start running.
launch.json (for Dart test runner) {
"name": "Dart: Run all tests",
"type": "dart",
"request": "launch",
"program": "./test/"
},
Timed test run:
00:05 +1 -7: Some tests failed.
real 0m9.532s
user 0m0.075s
sys 0m0.138s
I am using nested group blocks, but I don't believe this should cause such an enormous delay that makes TDD extremely slow
Example section of codevoid main() {
group('AuthCubit:', () {
late AuthCubit authCubit;
late MockLogIn logIn;
late MockLogOut logOut;
late MockRestoreSession restoreSession;
setUp(() {
logIn = MockLogIn();
logOut = MockLogOut();
restoreSession= MockRestoreSession();
authCubit = AuthCubit(
logIn : logIn,
logOut : logOut,
restoreSession: restoreSession,
);
});
tearDown(() {
authCubit.close();
});
test('initial state is unauthenticated', () async {
expect(authCubit.state, equals(const AuthStateUnauthenticated()));
});
group('logIn():', () {
//? Create late variables that all stages use, if needed
setUp(() {});
//? Create subgroups for stages this function will step through
group('On failure:', () {
setUp(() {});
test('emits unauthenticated state', () async { throw UnimplementedError(); });
});
group('On success:', () {
setUp(() {});
test('emits authenticated state', () async { throw UnimplementedError(); });
});
});
});
// ... + similar empty tests for other functions ...
}
I've tried even testing two empty example test files both separately and combined (see below code) both take 4+ seconds to run individually. If I combine both test files into one file, the execution time is the almost the same as just one test.
The problem seems to be around launching each test file, not the tests themselves.
import 'package:flutter_test/flutter_test.dart';
//? Create Mocks, Fakes, etc, if needed
void main() {
group('Test File 1:', () {
//? Create late variables that all functions use, if needed
test('empty test 1', () async { throw UnimplementedError(); });
test('empty test 2', () async { throw UnimplementedError(); });
test('empty test 3', () async { throw UnimplementedError(); });
test('empty test 4', () async { throw UnimplementedError(); });
test('empty test 5', () async { throw UnimplementedError(); });
});
}
import 'package:flutter_test/flutter_test.dart';
//? Create Mocks, Fakes, etc, if needed
void main() {
group('Test File 2:', () {
//? Create late variables that all functions use, if needed
test('empty test 1', () async { throw UnimplementedError(); });
test('empty test 2', () async { throw UnimplementedError(); });
test('empty test 3', () async { throw UnimplementedError(); });
test('empty test 4', () async { throw UnimplementedError(); });
test('empty test 5', () async { throw UnimplementedError(); });
});
}
import 'package:flutter_test/flutter_test.dart';
//? Create Mocks, Fakes, etc, if needed
void main() {
group('Test File 1:', () {
//? Create late variables that all functions use, if needed
test('empty test 1', () async { throw UnimplementedError(); });
test('empty test 2', () async { throw UnimplementedError(); });
test('empty test 3', () async { throw UnimplementedError(); });
test('empty test 4', () async { throw UnimplementedError(); });
test('empty test 5', () async { throw UnimplementedError(); });
});
group('Test File 2:', () {
//? Create late variables that all functions use, if needed
test('empty test 1', () async { throw UnimplementedError(); });
test('empty test 2', () async { throw UnimplementedError(); });
test('empty test 3', () async { throw UnimplementedError(); });
test('empty test 4', () async { throw UnimplementedError(); });
test('empty test 5', () async { throw UnimplementedError(); });
});
}
00:04 +0 -5: Some tests failed.
real 0m8.743s
user 0m0.060s
sys 0m0.167s
00:05 +0 -5: Some tests failed.
real 0m8.982s
user 0m0.137s
sys 0m0.106s
00:04 +0 -10: Some tests failed.
real 0m8.602s << Note this is actually FASTER than the smaller test files
user 0m0.045s << same ^
sys 0m0.200s
00:08 +0 -20: Some tests failed.
real 0m12.696s
user 0m0.015s << Weirdly, this is the smallest value out of all test cases
sys 0m0.152s
What could be the issue with my code, setup (software or hardware), with Dart/Flutter, or anything else?
Pretend I know nothing about Flutter, Dart, or VS Code; what questions would help find the potential causes and solutions for this?
jensjoha on GitHub helped me find a workaround to this by creating "nested aggregate test files"
On a higher level note I should probably add that for each file you have tests in, running tests are going to be slower because it has to do a compile, launch a new process etc. Similarly - as we saw - getting to the point where it actually starts doing something with the tests takes quite a while, so running flutter test specificfile.dart (at least for many specific files) isn't great. As somewhat of a workaround - when you want to run all tests - you could try to create one test file like done in #86722 (comment) and only run that.
// test_group_1_test.dart
import test_file_1_test.dart as testFile1;
import test_file_2_test.dart as testFile2;
// ...
void main() {
group("test file 1", testFile1.main);
group("test file 2", testFile2.main);
// ...
}
// everything_test.dart
import test_group_1_test.dart as testGroup1
import test_group_2_test.dart as testGroup2
// ...
void main() {
group("test group 1", testGroup1.main);
group("test group 2", testGroup2.main);
// ...
}
With this structure, I can run all (or any group of) test files within the same time it takes to run a single test file, which for my case is ~75% faster.