Code Coverage with pytest-cov

The Problem

You follow all the best practices so far. You write tests for your code. The application gets bigger and bigger. How do you know that you have tests for all the code?

Maybe you’ve missed some edge cases like:

def tricky_add(a: int, b: int) -> int | None:
    if a < 0:
        return None
    return a + b

i.e., you forgot to test the case when a is negative and we enter the if branch???

The Solution

Software engineers had those issues for a long time. One of the solutions is to measure the test coverage. So the percentage of the code that is covered by tests.

In the most basic form, you just check which lines are covered by tests and which are not. It’s not a silver bullet, but it’s a good indicator of how well the code is tested.

For example if you have line

x = foo(a) if a > 0 else bar(a)

and run it only with a > 0, you will not cover the bar(a) part.

However, if you see with test coverage that you didn’t cover this line, you know that neither foo nor bar are called here.

Test Coverage in Python

pytest-cov is a plugin for Pytest that provides test coverage reports. And if you combine them with ryanluker.vscode-coverage-gutters VS Code extension, you can see which lines are covered by tests and which are not directly in the editor!

Tapyr comes with those tools pre-configured.

Note

VS Code team works currently (April 2024) on adding test coverage directly to the editor. It’s already available with js tests. Check the March 2024 VS Code Release Notes for more information.

How to Use pytest-cov

First, because we defined

[tool.pytest.ini_options]
addopts = "--cov . --cov-report=lcov:lcov.info --cov-report=term"

in pyproject.toml, test coverage is already enabled and calculated every time you run the tests with pytest.

Second, you can turn on the coverage-gutters extension in VS Code to see the coverage directly in the editor.