Search code examples
dockerctestcdash

CDash timeout on code coverage submit

I am using the Kitware/cdash-docker docker compose files to build a docker container running CDash. I am running submissions from a large C++ project that generates around 3.5Mb XML files containing the code coverage information. ctest then times out on submit with the following message

Error when uploading file: <code coverage xml file name>
Error message was: Operation too slow. Less than 1 bytes/sec transferred the last 120 seconds
Problem when submitting via HTTP

I have modified the docker image to checkout the v2.4.0-prebuilt branch of CDash, and the problem remains on one docker host but when installed on another (less powerful machine) it actually accepts the submission and spends a little over a minute on the PUT request.

So my questions are, how can I modify the 120s limit for file submission, and are there settings for the CDash installation that may affect the performance for such uploads? I can't find anything in the docker container logs, so my understanding of the situation is that they keep doing their job until the client terminates on a timeout limit I can't figure out how to modify.

Solution

  • The submit timeout is hard coded into the CTest executable (as of cmake 3.20.3, in cmCTestSubmitHandler.cxx).

    The default value is 120 seconds. I patched that, increased it to 600s and my submissions are running. Need to investigate whether my server is struggling with something, or if this can be considered a CTest bug.

    Edit: The final and correct solution to this problem is to use asynchronous submissions. To use asynchronous submission, add the following line to config.local.php:

    $CDASH_ASYNCHRONOUS_SUBMISSION = true;

    This does not work from a port-mapped docker container, because an asynchronous submission creates a curl request to itself that it needs to start parsing the uploaded files. Since my CDash server was running on port 80 on the docker container which was mapped to port 8080 on the docker host, the curl request to itself inferred the usage of port 8080 which is incorrect from the docker container's point of view.

    Setting the CDash base URL to localhost explicitly fixes this problem

    $CDASH_BASE_URL = 'http://localhost';