How to find code which was never executed in coverage.py despite a 100% coverage report

Question

Consider the following code:

import math

def dumb_sqrt(x):
    result = math.sqrt(x) if x >= 0 else math.sqrt(-x)*j
    return result


def test_dumb_sqrt():
    assert dumb_sqrt(9.) == 3.

The test can be executed like this:

$ pip install pytest pytest-cov
$ pytest test_thing.py --cov=test_thing --cov-report=html --cov-branch

The coverage report will consider all lines 100% covered, even with branch coverage enabled:

However, this code has a bug, and those of you with a keen eye may have seen it already. Should it ever go into the "else" branch, there will be an exception:

NameError: global name 'j' is not defined

It's easy to fix the bug: change the undefined j name into a literal 1j. It's also easy to add another test which will reveal the bug: assert dumb_sqrt(-9.) == 3j. Neither is what this question is asking about. I want to know how to find sections of code which were never actually executed despite a 100% code coverage report.

Using conditional expressions is one such culprit, but there are similar cases anywhere that Python can short-circuit an evaluation (x or y, x and y are other examples).

Preferably, the line 4 above could be colored as yellow in the report, similar to how the "if" line would have rendered had it avoided using the conditional expression in the first place:

Does coverage.py support such a feature? If so, how can you enable "inline branch coverage" in your cov reporting? If not, are there any other approaches to identify "hidden" code that was never actually executed by your test suite?

Answer 1

No, coverage.py doesn't handle conditional branching within an expression. This doesn't just affect the Python conditional expression, using and or or would also be affected:

# pretending, for the sake of illustration, that x will never be 0
result = x >= 0 and math.sqrt(x) or math.sqrt(-x)*j

Ned Batchelder, the maintainer of coverage.py, calls this a hidden conditional, in an article from 2007 covering this and other cases coverage.py can't handle.

This problem extends to if statements too! Take, for example:

if condition_a and (condition_b or condtion_c):
    do_foo()
else:
    do_bar()

If condition_b is always true when condition_a is true, you'll never find the typo in condtion_c, not if you rely solely on converage.py, because there is no support for conditional coverage (let alone more advanced concepts like modified condition/decision coverage and multiple condition coverage.

One hurdle to supporting conditional coverage is technical: coverage.py relies heavily on Python's built-in tracing support, but until recently this would only let you track execution per line. Ned actually explored work-arounds for this issue.

Not that that stopped a different project, instrumental from offering condition/decision coverage anyway. That project used AST rewriting and an import hook to add extra bytecode that'd let it track the outcome of individual conditions and so give you an overview of the 'truth table' of expressions. There is a huge drawback to this approach: it is very fragile, and frequently requires updating for new Python releases. As a result, the project broke with Python 3.4 and hasn't been fixed.

However, Python 3.7 added support for tracing at the opcode level, allowing a tracer to analyse the effect of each individual bytecode without having to resort to AST hacking. And with coverage.py 5.0 having reached a stable state, it appears that the project is considering adding support for condition coverage, with possible sponsors to back development.

So your options right now are:

• Run your code in Python 3.3 and use instrumental
• Repair instrumental to run on more recent Python versions
• Wait for coverage.py to add condition coverage
• Help write the feature for coverage.py
• Hack together your own version of instrumental's approach using the Python 3.7 or newer 'opcode' tracing mode.