Search code examples
angularcypressistanbulnyc

Merging the Code coverage report JSON files from Cypress and Karma does not produce a correct result

I have an angular project, which I want to test with the built in unit testing tool (karma) and with cypress as well. My task is to merge the code coverage reports of both.

I have created successfully the coverage configurations. The outputs are merged with nyc merge successfully: however, the numbers don't add up.

Related versions from package.json:

    "@angular/core": "^16.0.3",
    "cypress": "^12.13.0",
    "istanbul-lib-coverage": "^3.2.0",
    "@cypress/code-coverage": "^3.11.0",
    "@cypress/webpack-preprocessor": "^5.17.1",
    "@istanbuljs/nyc-config-typescript": "^1.0.2",
    "@jsdevtools/coverage-istanbul-loader": "^3.0.5",
    "karma": "^6.4.2",
    "karma-coverage": "^2.2.0",
    "nyc": "^15.1.0",

Here is an example of a file in the result of the Karma coverage report in coverage-final.json:

...
".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {"path":".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts","statementMap":{"0":{"start":{"line":16,"column":11},"end":{"line":16,"column":38}},"1":{"start":{"line":17,"column":12},"end":{"line":17,"column":40}},"2":{"start":{"line":18,"column":21},"end":{"line":18,"column":45}},"3":{"start":{"line":19,"column":12},"end":{"line":19,"column":38}},"4":{"start":{"line":23,"column":4},"end":{"line":23,"column":56}},"5":{"start":{"line":27,"column":4},"end":{"line":27,"column":30}},"6":{"start":{"line":28,"column":4},"end":{"line":28,"column":35}},"7":{"start":{"line":29,"column":4},"end":{"line":29,"column":62}},"8":{"start":{"line":33,"column":4},"end":{"line":33,"column":39}},"9":{"start":{"line":37,"column":4},"end":{"line":37,"column":30}},"10":{"start":{"line":38,"column":4},"end":{"line":38,"column":31}},"11":{"start":{"line":11,"column":13},"end":{"line":40,"column":null}}},"fnMap":{"0":{"name":"(anonymous_0)","decl":{"start":{"line":15,"column":2},"end":{"line":15,"column":null}},"loc":{"start":{"line":19,"column":38},"end":{"line":20,"column":6}}},"1":{"name":"(anonymous_1)","decl":{"start":{"line":22,"column":2},"end":{"line":22,"column":10}},"loc":{"start":{"line":22,"column":10},"end":{"line":24,"column":3}}},"2":{"name":"(anonymous_2)","decl":{"start":{"line":26,"column":2},"end":{"line":26,"column":16}},"loc":{"start":{"line":26,"column":22},"end":{"line":30,"column":3}}},"3":{"name":"(anonymous_3)","decl":{"start":{"line":32,"column":2},"end":{"line":32,"column":9}},"loc":{"start":{"line":32,"column":13},"end":{"line":34,"column":3}}},"4":{"name":"(anonymous_4)","decl":{"start":{"line":36,"column":2},"end":{"line":36,"column":18}},"loc":{"start":{"line":36,"column":18},"end":{"line":39,"column":3}}}},"branchMap":{},"s":{"0":11,"1":11,"2":11,"3":11,"4":7,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":1},"f":{"0":11,"1":7,"2":0,"3":0,"4":0},"b":{}}
...

The lcov-report of the Karma run: HTML report of the Karma run

The same file in the result of Cypress coverage in coverage-final.json:

...
".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {"path":".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts","statementMap":{"0":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}},"1":{"start":{"line":19,"column":38},"end":{"line":19,"column":null}},"2":{"start":{"line":16,"column":11},"end":{"line":16,"column":38}},"3":{"start":{"line":17,"column":12},"end":{"line":17,"column":40}},"4":{"start":{"line":18,"column":21},"end":{"line":18,"column":45}},"5":{"start":{"line":19,"column":12},"end":{"line":19,"column":38}},"6":{"start":{"line":23,"column":4},"end":{"line":23,"column":null}},"7":{"start":{"line":27,"column":4},"end":{"line":27,"column":null}},"8":{"start":{"line":28,"column":4},"end":{"line":28,"column":null}},"9":{"start":{"line":29,"column":4},"end":{"line":29,"column":null}},"10":{"start":{"line":33,"column":4},"end":{"line":33,"column":null}},"11":{"start":{"line":37,"column":4},"end":{"line":37,"column":null}},"12":{"start":{"line":38,"column":4},"end":{"line":38,"column":null}},"13":{"start":{"line":11,"column":13},"end":{"line":11,"column":null}}},"fnMap":{"0":{"name":"(anonymous_1)","decl":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}},"loc":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}}},"1":{"name":"(anonymous_2)","decl":{"start":{"line":15,"column":2},"end":{"line":15,"column":null}},"loc":{"start":{"line":19,"column":38},"end":{"line":20,"column":null}}},"2":{"name":"(anonymous_3)","decl":{"start":{"line":22,"column":2},"end":{"line":22,"column":10}},"loc":{"start":{"line":22,"column":10},"end":{"line":24,"column":null}}},"3":{"name":"(anonymous_4)","decl":{"start":{"line":26,"column":2},"end":{"line":26,"column":16}},"loc":{"start":{"line":26,"column":22},"end":{"line":30,"column":null}}},"4":{"name":"(anonymous_5)","decl":{"start":{"line":32,"column":2},"end":{"line":32,"column":9}},"loc":{"start":{"line":32,"column":13},"end":{"line":34,"column":null}}},"5":{"name":"(anonymous_6)","decl":{"start":{"line":36,"column":2},"end":{"line":36,"column":18}},"loc":{"start":{"line":36,"column":18},"end":{"line":39,"column":null}}}},"branchMap":{"0":{"loc":{"start":{"line":11,"column":13},"end":{"line":11,"column":38}},"type":"binary-expr","locations":[{"start":{"line":11,"column":13},"end":{"line":11,"column":38}},{"start":{"line":11,"column":13},"end":{"line":11,"column":38}}]}},"s":{"0":30,"1":190,"2":38,"3":38,"4":38,"5":38,"6":38,"7":0,"8":0,"9":0,"10":9992,"11":0,"12":0,"13":98},"f":{"0":30,"1":38,"2":38,"3":0,"4":9992,"5":0},"b":{"0":[38,38]}}
...

The lcov-report of the Cypress run: HTML report of the Cypress run

After merging the two with nyc merge I get this result `:

  ".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {
    "path": ".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts",
    "statementMap": {
      "0": {
        "start": {
          "line": 11,
          "column": 38
        },
        "end": {
          "line": 11,
          "column": null
        }
      },
      "1": {
        "start": {
          "line": 19,
          "column": 38
        },
        "end": {
          "line": 19,
          "column": null
        }
      },
      "2": {
        "start": {
          "line": 16,
          "column": 11
        },
        "end": {
          "line": 16,
          "column": 38
        }
      },
      "3": {
        "start": {
          "line": 17,
          "column": 12
        },
        "end": {
          "line": 17,
          "column": 40
        }
      },
      "4": {
        "start": {
          "line": 18,
          "column": 21
        },
        "end": {
          "line": 18,
          "column": 45
        }
      },
      "5": {
        "start": {
          "line": 19,
          "column": 12
        },
        "end": {
          "line": 19,
          "column": 38
        }
      },
      "6": {
        "start": {
          "line": 23,
          "column": 4
        },
        "end": {
          "line": 23,
          "column": null
        }
      },
      "7": {
        "start": {
          "line": 27,
          "column": 4
        },
        "end": {
          "line": 27,
          "column": null
        }
      },
      "8": {
        "start": {
          "line": 28,
          "column": 4
        },
        "end": {
          "line": 28,
          "column": null
        }
      },
      "9": {
        "start": {
          "line": 29,
          "column": 4
        },
        "end": {
          "line": 29,
          "column": null
        }
      },
      "10": {
        "start": {
          "line": 33,
          "column": 4
        },
        "end": {
          "line": 33,
          "column": null
        }
      },
      "11": {
        "start": {
          "line": 37,
          "column": 4
        },
        "end": {
          "line": 37,
          "column": null
        }
      },
      "12": {
        "start": {
          "line": 38,
          "column": 4
        },
        "end": {
          "line": 38,
          "column": null
        }
      },
      "13": {
        "start": {
          "line": 11,
          "column": 13
        },
        "end": {
          "line": 11,
          "column": null
        }
      }
    },
    "fnMap": {
      "0": {
        "name": "(anonymous_1)",
        "decl": {
          "start": {
            "line": 11,
            "column": 38
          },
          "end": {
            "line": 11,
            "column": null
          }
        },
        "loc": {
          "start": {
            "line": 11,
            "column": 38
          },
          "end": {
            "line": 11,
            "column": null
          }
        }
      },
      "1": {
        "name": "(anonymous_2)",
        "decl": {
          "start": {
            "line": 15,
            "column": 2
          },
          "end": {
            "line": 15,
            "column": null
          }
        },
        "loc": {
          "start": {
            "line": 19,
            "column": 38
          },
          "end": {
            "line": 20,
            "column": null
          }
        }
      },
      "2": {
        "name": "(anonymous_3)",
        "decl": {
          "start": {
            "line": 22,
            "column": 2
          },
          "end": {
            "line": 22,
            "column": 10
          }
        },
        "loc": {
          "start": {
            "line": 22,
            "column": 10
          },
          "end": {
            "line": 24,
            "column": null
          }
        }
      },
      "3": {
        "name": "(anonymous_4)",
        "decl": {
          "start": {
            "line": 26,
            "column": 2
          },
          "end": {
            "line": 26,
            "column": 16
          }
        },
        "loc": {
          "start": {
            "line": 26,
            "column": 22
          },
          "end": {
            "line": 30,
            "column": null
          }
        }
      },
      "4": {
        "name": "(anonymous_5)",
        "decl": {
          "start": {
            "line": 32,
            "column": 2
          },
          "end": {
            "line": 32,
            "column": 9
          }
        },
        "loc": {
          "start": {
            "line": 32,
            "column": 13
          },
          "end": {
            "line": 34,
            "column": null
          }
        }
      },
      "5": {
        "name": "(anonymous_6)",
        "decl": {
          "start": {
            "line": 36,
            "column": 2
          },
          "end": {
            "line": 36,
            "column": 18
          }
        },
        "loc": {
          "start": {
            "line": 36,
            "column": 18
          },
          "end": {
            "line": 39,
            "column": null
          }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": {
            "line": 11,
            "column": 13
          },
          "end": {
            "line": 11,
            "column": 38
          }
        },
        "type": "binary-expr",
        "locations": [
          {
            "start": {
              "line": 11,
              "column": 13
            },
            "end": {
              "line": 11,
              "column": 38
            }
          },
          {
            "start": {
              "line": 11,
              "column": 13
            },
            "end": {
              "line": 11,
              "column": 38
            }
          }
        ]
      }
    },
    "s": {
      "0": 183,
      "1": 983,
      "2": 245,
      "3": 245,
      "4": 233,
      "5": 212,
      "6": 190,
      "7": 0,
      "8": 0,
      "9": 0,
      "10": 49960,
      "11": 3,
      "12": 0,
      "13": 490,
      "14": null,
      "15": null,
      "16": null,
      "17": null,
      "18": null,
      "19": null,
      "20": null,
      "21": null
    },
    "f": {
      "0": 183,
      "1": 211,
      "2": 190,
      "3": 0,
      "4": 49960,
      "5": 0,
      "6": null,
      "7": null,
      "8": null,
      "9": null,
      "10": null
    },
    "b": {
      "0": [
        190,
        190
      ]
    }
  }

Running then nyc report --reporter=lcov --reporter=text-summary results in:

merged report

What can be the problem here? I have tried some of the recommendations in the https://github.com/istanbuljs/nyc/issues/1302 here, but non of them worked. Moving to earlier versions of Istambul (e.g. 14.1) produces gibberish data (it won't parse my Typescript files).

Update: Here is my command to generate the outputs:

  1. first, I validate that I have an existing reports in the karma and cypress folders.
  2. command:
    "coverage-merge-and-report": "mkdir -p coverage/toMerge && cp coverage/karma/coverage-final.json coverage/toMerge/ && cp coverage/cypress/coverage-final.json coverage/toMerge/coverage-final-cypress.json && nyc merge coverage/toMerge coverage/toMerge/merged.json && npx nyc report --reporter=lcov --reporter=text-summary --temp-dir=coverage/toMerge --report-dir=coverage/merged-report"

The problem with it was with the final part: npx nyc report --reporter=lcov --reporter=text-summary --report-dir=... scans for the .nyc_output folder I belive, and the reportes is not right either. enter image description here

I changed it to this: npx nyc report --reporter=html --reporter=text-summary -t coverage/test --report-dir=... and now it produces a correct answer.

Solution

  • If I take your karma and cypress outputs, put them in a clean project in /cover folder and run this statement (commented line is the syntax)

    // nyc merge <input-directory> [output-file]
   nyc merge cover cover-merged.json

    I don't get the same merged file as yours.

    Spot checking

    • in karma statementMap (index:0) is line 16 and it has a count of 11
    • in cypress statementMap line 16 is in index:2 and has count 38
    • my merged output also has line 16 in index:2 and has count 49

    Since your numbers are way larger, are there some other files getting dragged into the merge?

    Also the line 16 count in cypress lcov-report is 147 but in cypress coverage-final.json is 38.

    I would concentrate on comparing the json files first off. There aren't any other clues in the question, but check your parameters for the nyc merge <input-directory> [output-file] command, and look for prior-run output files that could be bumping up the counts.

    VS Code side-by-side

    enter image description here