Flutter golden image tests diff threshold

Question

I am making use of golden image tests for simple stateless widgets in flutter. Every few test runs, I get the following error printed to console:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure object was thrown running a test (but after the test had completed):
  Expected: one widget whose rasterized image matches golden image
"goldens/mywidget.png"
  Actual: _WidgetTypeFinder:<zero widgets with type "MyWidget" (ignoring offstage widgets)>
   Which: Golden "goldens/mywidget.png": Pixel test failed, 0.04% diff detected.
          Failure feedback can be found at
/home/cirrus/project/test/widgets/failures

When the exception was thrown, this was the stack:

This is infuriating for 2 reasons:

1. Nothing has changed, the diffs appear to be subpixel rendering differences. This shouldn't be happening, and is thus rendering golden tests pretty useless.
2. Even with the message like the above, the test passes. So flutter thinks there's an exception, but doesn't even fail the test for it.

Is there a way to set a diff threshold so that tests with a diff below 5% for example are ignored, to avoid my test output being overly verbose and noisy?

It is worth noting that this issue is not isolated to our CI server, it happens on our local machines too, so seems odd that the same machine would render the test differently depending on the time of day.

Answer 1

I encountered the same problem, and was looking for a similar solution. After discussion on a Github issue I created, I learned a way to set the threshold.

First, create a file somewhere in your test folder (I put it in test/utils), say local_file_comparator_with_threshold.dart:

    import 'dart:typed_data';
    
    import 'package:flutter/foundation.dart';
    import 'package:flutter_test/flutter_test.dart';
    
    /// Works just like [LocalFileComparator] but includes a [threshold] that, when
    /// exceeded, marks the test as a failure.
    class LocalFileComparatorWithThreshold extends LocalFileComparator {
      /// Threshold above which tests will be marked as failing.
      /// Ranges from 0 to 1, both inclusive.
      final double threshold;
    
      LocalFileComparatorWithThreshold(Uri testFile, this.threshold)
          : assert(threshold >= 0 && threshold <= 1),
            super(testFile);
    
      /// Copy of [LocalFileComparator]'s [compare] method, except for the fact that
      /// it checks if the [ComparisonResult.diffPercent] is not greater than
      /// [threshold] to decide whether this test is successful or a failure.
      @override
      Future<bool> compare(Uint8List imageBytes, Uri golden) async {
        final result = await GoldenFileComparator.compareLists(
          imageBytes,
          await getGoldenBytes(golden),
        );
    
        if (!result.passed && result.diffPercent <= threshold) {
          debugPrint(
            'A difference of ${result.diffPercent * 100}% was found, but it is '
            'acceptable since it is not greater than the threshold of '
            '${threshold * 100}%',
          );
    
          return true;
        }
    
        if (!result.passed) {
          final error = await generateFailureOutput(result, golden, basedir);
          throw FlutterError(error);
        }
        return result.passed;
      }
    }

And then, you can override your test configuration in flutter_test_config.dart (you need to put in your project's test folder, create one if you don't already have one):

import 'dart:async';

import 'package:flutter_test/flutter_test.dart';

import 'utils/local_file_comparator_with_threshold.dart';

/// Customise your threshold here
/// For example, the error threshold here is 0.5%
/// Golden tests will pass if the pixel difference is equal to or below 0.5%
const _kGoldenTestsThreshold = 0.5 / 100;

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  if (goldenFileComparator is LocalFileComparator) {
    final testUrl = (goldenFileComparator as LocalFileComparator).basedir;

    goldenFileComparator = LocalFileComparatorWithThreshold(
      // flutter_test's LocalFileComparator expects the test's URI to be passed
      // as an argument, but it only uses it to parse the baseDir in order to
      // obtain the directory where the golden tests will be placed.
      // As such, we use the default `testUrl`, which is only the `baseDir` and
      // append a generically named `test.dart` so that the `baseDir` is
      // properly extracted.
      Uri.parse('$testUrl/test.dart'),
      _kGoldenTestsThreshold,
    );
  } else {
    throw Exception(
      'Expected `goldenFileComparator` to be of type `LocalFileComparator`, '
      'but it is of type `${goldenFileComparator.runtimeType}`',
    );
  }

  await testMain();
}

After this setup, whenever you run your golden tests in the project, they will pass when the pixel difference is less than or equal to the threshold you set in flutter_test_config.dart.