Search code examples
cmakectestcdash

Upload image diff using CTest and CDash


For running automated tests in a C++ application, I would like the application to dump an image and compare it against a baseline image. I saw several examples of this on various CDash dashboards, e.g. this one (link might not be valid for long). https://open.cdash.org/testDetails.php?test=660365465&build=5407474

My google-fu has failed me on this one, what is the correct way to get this functionality?


Solution

  • The easiest way to attach ordinary files to test results is by listing those files in either the ATTACHED_FILES or ATTACHED_FILES_ON_FAIL test properties. This is not the mechanism being used here though.

    According to this mailing list post, you can output special contents like that shown below to the test's stdout and it results in the named files being uploaded. The sample CDash results page you linked to follows a similar pattern as the example from the mailing list, which I've reproduced here for reference (I've made one small correction to change DifferenceImage to DifferenceImage2):

    <DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement>
    <DartMeasurementFile name="TestImage" type="image/png">C:/Users/.../Testing/Temporary/BoxWidget.png</DartMeasurementFile>
    <DartMeasurementFile name="DifferenceImage2" type="image/png">C:/Users/.../Testing/Temporary/BoxWidget.diff.png</DartMeasurementFile>
    <DartMeasurementFile name="ValidImage" type="image/png">C:/Users/.../VTKData/Baseline/Widgets/BoxWidget.png</DartMeasurementFile>
    

    I've checked through the CTest source code and it scans the test output looking for <DartMeasurement> and <DartMeasurementFile> tags here and here. These are uploaded as discrete measurement items to CDash, which also looks for these particular names and presents them specially as in the example CDash links in the question.