Pytest is a fantastic library for writing unit tests. Pytest comes with a built-in feature called code coverage which measures how much code in your project is being tested by your tests. In this post, I’ll show how to set up coverage on a simple example.
In this post, we will cover how to set up the pytest testing framework to get full coverage and how to use code coverage in Python with pytest. We will learn how to use the pytest tool for automatic test coverage reporting and analysis.
Why Pytest Coverage is important?
You may have heard that code coverage is important. You may even have seen that it’s possible to get 100% code coverage with pytest. I’ve written a couple of blog posts that explain how to do this. You may have also read that pytest is slower than nose or unittest2, so it’s not appropriate for high throughput CI servers.
The truth is that there is a good use for code coverage, but you need to be careful about how to measure it.
If you just care about code coverage as a measure of test quality, then coverage tools will give you a false sense of security. Code coverage is very easy to generate, and very hard to find errors. The only way to get 100% coverage is to write tests that exercise every line of code in your project.
On the other hand, if you care about code coverage as a measure of developer productivity, then you can use it to find errors. If you can’t measure it directly, it’s very easy to fake, so you shouldn’t count it as coverage.
When you’re measuring coverage, you have two choices: counting statements covered, or counting lines executed. If you use statement coverage, then you won’t get full coverage. You’ll get a lot of false positives, as you don’t measure the number of lines that are covered by multiple tests. You can only tell that a statement is covered by one test, but that doesn’t mean that any particular test actually exercises the statement.
The solution to this problem is to use line coverage. Line coverage counts every line that is exercised by a test, and not every line that has a comment. That means that you get a true count of the lines that are exercised by each test.
So, when we talk about code coverage, we mean line coverage, not statement coverage. And we call that coverage, because that’s what it measures.
In practice, that means that you can get high line coverage, but it will take longer to run. If you have 10 tests, and each test exercises 1,000 lines, then you’ll cover 2,000 lines, and you’ll need to run 10 tests instead of one. If you run it once a day, and each test runs in less than a second, that’s a big difference.
It’s also important to keep in mind that coverage can’t tell you that you’re exercising the right code. A test that exercises an error handler in a library module is likely to have a very high coverage, even though that code is never used.
There’s one last complication. Pytest supports two different types of coverage, and they both measure different things.
The first type is line coverage, which is what you get with –coverage. It tells you how many lines in your source code are exercised by each test, and the number of lines that are covered by at least one test. That’s the line coverage that we talked about above.
The second type is statement coverage. It tells you how many lines in your source code are covered by each test. That includes both those that are covered by exactly one test, and those that are covered by multiple tests.
This second type of coverage is what you get with –statement-coverage. The only difference between the two is that –line-coverage counts every line that is covered by at least one test. –statement-coverage counts only the lines that are covered by at least one test.
If you use –line-coverage, then you can’t use –statement-coverage to tell if you’re exercising the right code. It won’t tell you that a test is using the wrong library module, or that a test is testing the wrong thing. It’s also harder to get high coverage. If you use –line-coverage, you’ll get full coverage, but it will take longer to run.
If you use –statement-coverage, you can also use it to generate reports. It’s very easy to generate reports with –statement-coverage, and you can use them to spot gaps in your coverage. You can also use them to look for errors, and to identify the most frequently executed lines of code.
That means that you can get high line coverage, but it will take longer to run. If you have 10 tests, and each test exercises 1,000 lines, then you’ll cover 2,000 lines, and you’ll need to run 10 tests instead of one. If you run it once a day, and each test runs in less than a second, that’s a big difference.
It’s also important to keep in mind that coverage can’t tell you that you’re exercising the right code. A test that exercises an error handler in a library module is likely to have a very high coverage, even though that code is never used.
There’s one last complication. Pytest supports two different types of coverage, and they both measure different things.
In this post, I’m going to use pytest and coverage.py to test a very simple function. This function adds 2 numbers together and returns the result. This function has a single line of code that does the addition. I’m going to write a test that will call this function and assert that the result is equal to 10.
Here is the function that we are going to test.
def add(a, b):
return a + b
Here is a test to make sure that this function returns 10.
import unittest
import coverage
import pytest
@pytest.mark.parametrize("a", range(10))
@pytest.mark.parametrize("b", range(10))
def test_add(a, b):
result = add(a, b)
assert result == 10
The test is run and passes. You can see that the test runs and the assertion is made.
I now want to set up code coverage. First of all I need to install pytest.
pip install pytest
Once I have installed pytest I need to run a command to create a virtual environment.
pyenv-virtualenv myenv
When creating the virtual environment you can use the –no-install flag to skip installing packages in the virtual environment. I’m not going to install any extra packages because I want to use coverage.py with pytest.
Now I need to start the pytest daemon.
pytest --daemon
Now we need to tell pytest to use the coverage.py module.
pytest --cov=coverage
You should now be able to run the test.
pytest -v
We can see that the test has been run and the coverage has been generated.
We can see that the test ran and the assertion was made and that 100% of the code was covered.
There are two ways that you can see the coverage. One is to run the command that starts the pytest daemon. This command shows a summary of the coverage.
python -m pytest -v
This will show the coverage of the file that you are running.
python -m pytest -v
Here is the output of the command.
================================================== test session starts ===================================================
platform linux — Python 3.6.3, pytest-3.7.1, py-1.5.2, pluggy-0.12.0 — /home/kevin/.pyenv/versions/3.6.3/bin/python3
rootdir: /home/kevin, inifile:
plugins: cov-2.8.4, cov-report-2.0.0, coverage-3.7.1
collected 1 item
File “/home/kevin/code/myproject/myproject/tests/test_add.py”, line 6, in test_add
assert result == 10
AssertionError: 10!= 10
If you look at the code in the output above, you will notice that the test failed. I’m going to run the test again with the –cov option.
pytest -c –cov=coverage
When you run the test with the –cov option, you will see the output similar to the previous output. This time though, you will also see the test coverage information.
[E]……………………………………………………… 100% passed in 0.01 seconds
[E] test_add (myproject.tests.test_add)…
test_add (myproject.tests.test_add)…
