QTest executes test case twice

Question

I've written a small benchmark in QTest, and although I've used QBENCHMARK_ONCE.

Here some example code replicating the issue:

header:

#ifndef MY_TEST_H
#define MY_TEST_H

#include <QtTest>

class MyTest : public QObject
{
    Q_OBJECT

private slots:
    void initTestCase();
    void test1();
};

#endif // MY_TEST_H

cpp file:

#include "mytest.h"

void MyTest::initTestCase() {
    qDebug() << "init";
}

void MyTest::test1() {
    QBENCHMARK_ONCE {
        qDebug() << "bench";
    }

    qDebug() << "test1";
}

QTEST_MAIN(MyTest)

Running "mytest" I get:

********* Start testing of MyTest *********
Config: Using QtTest library 5.5.1, Qt 5.5.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 5.3.1 20160407)
QDEBUG : MyTest::initTestCase() init
PASS   : MyTest::initTestCase()
QDEBUG : MyTest::test1() bench
QDEBUG : MyTest::test1() test1
QDEBUG : MyTest::test1() bench
QDEBUG : MyTest::test1() test1
PASS   : MyTest::test1()
RESULT : MyTest::test1():
     0 msecs per iteration (total: 0, iterations: 1)
PASS   : MyTest::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted
********* Finished testing of MyTest *********

I would like it to run once and only once. Some of the benchmarks take one Minute per iteration...

I'm using CMake with make back-end on Linux. They test classes get compiled into separate executables. Since ctest doesn't give me useful output, I run them directly, i.e., "# ./mytest"

//Addition: CMakeLists.txt

include(CTest)
enable_testing()
set(CMAKE_AUTOMOC ON)
find_package(Qt5Test REQUIRED)

add_executable(mytest "test/mytest.cpp")
add_test(mytest mytest)
target_link_libraries(mytest Qt5::Test)

Answer 1

Apparently, this is intended behavior in order to mitigate the impact of initialization and caches on the result:

https://bugreports.qt.io/browse/QTBUG-12689

I think they should add that to the documentation though. Or name the macro less confusingly.