More metrics for CodeCoverage Elixir

Question

Background

I have a test suite and I need to know the coverage of the project. I have played around with mix test --cover but I find the native erlang's coverage analysis tool to be insufficient at best.

The native coverage tool doesn't tell you about branch coverage nor function coverage. It's only metric seems to be relevant lines which I have no idea how they calculate. For all I know, this is just the most basic form of test coverage: see if a given text line was executed.

What have you tried?

I have tried Coverex but the result was disastrous. Not only does it suffer from the same issues that the native tool does, it also seems not produce correct results as it counts imported modules as untested.

Or maybe it is doing a great job and my code is poorly tested, but I can't know for sure because it doesn't tell me how it is evaluating my code. Have 40% coverage in a file? What am I missing? I can't know, the tool wont tell me.

I am now using ExCoveralls. It is considerably better than the previous options, it allows me to easily configure which folders I want to ignore, but it uses the native coverage tool, so it suffers pretty much from the same issues.

What do you want?

I was hoping to find something among the lines of Istanbul, or in this case nyc:

https://github.com/istanbuljs/nyc

It's test coverage analysis tells me everything I need to know, metrics and all:

Branches, Functions, Lines, Statements, everything you need to know is there.

Questions

1. Is there any tool that uses Istanbul for code coverage metrics with Elixir instead of the native erlang one?
2. If not, is there a way to configure the native coverage tool to give me more information?
3. Which metrics does the native coverage tool uses ?

Answer 1

The native coverage tool inserts "bump" calls on every line of the source code, recording module, function, arity, clause number and line number:

bump_call(Vars, Line) ->
    A = erl_anno:new(0),
    {call,A,{remote,A,{atom,A,ets},{atom,A,update_counter}},
     [{atom,A,?COVER_TABLE},
      {tuple,A,[{atom,A,?BUMP_REC_NAME},
                {atom,A,Vars#vars.module},
                {atom,A,Vars#vars.function},
                {integer,A,Vars#vars.arity},
                {integer,A,Vars#vars.clause},
                {integer,A,Line}]},
      {integer,A,1}]}.

(from cover.erl)

The code inserted by the function above is:

ets:update_counter(?COVER_TABLE,
  {?BUMP_REC_NAME, Module, Function, Arity, Clause, Line}, 1)

That is, increment the entry for the given module / function / line in question by 1. After all tests have finished, cover will use the data in this table and show how many times a given line was executed.

---

As mentioned in the cover documentation, you can get coverage for modules, functions, function clauses and lines. It looks like ExCoveralls only uses line coverage in its reports, but there is no reason it couldn't do all four types of coverage.

Branch coverage is not supported. Seems like supporting branch coverage would require expanding the "bump" record and updating cover.erl to record that information. Until someone does that, coverage information is only accurate when branches appear on different lines. For example:

case always_false() of
    true ->
        %% this line shows up as not covered
        do_something();
    false ->
        ok
end.

%% this line shows up as covered, even though do_something is never called
always_false() andalso do_something()