Karma-coverage exit code is always 0 even when test coverage threshold is not met

Question

I'm trying to build a gitlab-ci pipeline for UI5 Apps to include a test stage using Karma runner and karma-ui5. I would like the pipeline to fail if test coverage threshold are not met. My karma.conf.js looks like this:

module.exports = function(config) {
    "use strict";

    config.set({

        frameworks: ["ui5"],

        preprocessors: {
            "{webapp,webapp/!(test)}/*.js": ["coverage"]
        },

        coverageReporter: {
            includeAllSources: true,
            reporters: [
                {
                    type: "html",
                    dir: "coverage"
                },
                {
                    type: "text"
                }
            ],
            check: {
                each: {
                    statements: 100,
                    branches: 100,
                    functions: 100,
                    lines: 100
                }
            }
        },

        reporters: ["progress", "coverage"],

        browsers: ["CustomChromeHeadless"],
        customLaunchers: {
            CustomChromeHeadless: {
                base: 'ChromeHeadless',
                flags: ['--no-sandbox']
            }
        },

        browserConsoleLogOptions: {
            level: "error"
        },

        singleRun: true
    });
};

Karma coverage reports the files that do not meet coverage threshold but the exit code is always 0, thus the pipeline does not fail.

The output looks like this:

TOTAL: 2 SUCCESS
07 11 2020 10:08:46.033:ERROR [coverage]: HeadlessChrome 64.0.3282 (Linux 0.0.0): Coverage for branches (50%) does not meet per-file (/builds/younang/ci/web/webapp/controller/View.controller.js)  threshold (100%)
07 11 2020 10:08:46.034:ERROR [coverage]: HeadlessChrome 64.0.3282 (Linux 0.0.0): Coverage for statements (0%) does not meet per-file (/builds/younang/ci/web/webapp/localService/mockserver.js)  threshold (100%)
07 11 2020 10:08:46.034:ERROR [coverage]: HeadlessChrome 64.0.3282 (Linux 0.0.0): Coverage for branches (0%) does not meet per-file (/builds/younang/ci/web/webapp/localService/mockserver.js)  threshold (100%)
07 11 2020 10:08:46.034:ERROR [coverage]: HeadlessChrome 64.0.3282 (Linux 0.0.0): Coverage for lines (0%) does not meet per-file (/builds/younang/ci/web/webapp/localService/mockserver.js)  threshold (100%)
07 11 2020 10:08:46.034:ERROR [coverage]: HeadlessChrome 64.0.3282 (Linux 0.0.0): Coverage for functions (0%) does not meet per-file (/builds/younang/ci/web/webapp/localService/mockserver.js)  threshold (100%)
-------------------|---------|----------|---------|---------|-------------------
File               | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-------------------|---------|----------|---------|---------|-------------------
All files          |      25 |     4.35 |   41.18 |      25 |                   
 webapp            |     100 |      100 |     100 |     100 |                   
  Component.js     |     100 |      100 |     100 |     100 |                   
 webapp/controller |     100 |       50 |     100 |     100 |                   
  ...controller.js |     100 |       50 |     100 |     100 | 16                
 ...p/localService |       0 |        0 |       0 |       0 |                   
  mockserver.js    |       0 |        0 |       0 |       0 | 1-115             
 webapp/model      |     100 |      100 |     100 |     100 |                   
  models.js        |     100 |      100 |     100 |     100 |                   
-------------------|---------|----------|---------|---------|------------------

You can see the full setup in Gitlab here

Any idea what am i missing here? Help much appreciated.

Thanks

Answer 1

Looks like this is a known issue. The solution there was to upgrade karma to 5.2.0. Maybe give that a shot?