Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Testing in Python

  • How can we make sure our code in Python works as expect?
  • How can we make sure changes in one area do not impact other area?
  • How can we upgrade dependencies and makes sure everything works as before?
  • How can we switch to another version of Python without fear?
  • How can we change or upgrade the Operating System where our code runs and know it still functions?

The software testing equation

INPUT + PROCESS = EXPECTED_OUTPUT

The software testing equation (fixed)

INPUT + PROCESS = EXPECTED_OUTPUT + BUGS

Manual testing

  • How do you check your application now?

This is the place of a discussion. If the time and environment permits.

Traditional Organizations

In traditional organizations the software development process looks something like this:

  1. Months of planning.
  2. Many months of development.
  3. Many months of testing / qa (Quality Assurance) and waiting for the development team to fix the issues.
  4. Release once every few months or once a year.

This method was described as the Waterfall process.

Quality Assurance

  • Nightly build.
  • Testing new features.
  • Testing bug fixes.
  • Maybe testing critical features again and again…
  • …or maybe not.
  • Regression testing?
  • Testing / manual qa has a huge boring repetative part.
  • It is also very slow and expensive.

Web age Organizations

  • Very frequent releases (20-30 / day!) (Amazon is said to have a new release of some part of their stack every second!)
  • Very little time for manual testing.
  • The solution is to use CI/CD that stand for:
    • CI - Continuous Integration
    • CD - Continuous Delivery
    • CD - Continuous Deployment

TDD vs Testing as an Afterthought

TDD - Test Driven Development.

It is an excellent approach but very few organizations manage to implement it. It requires a lot of discipline both by the development team and by the management.

Testing as an afterthought

This is what happens in most of the organizations. Even in start-ups. They want to develop the product as fast as possible and they cut corners where possible.

After a while you reach the point where

  • There is an existing product (service).
  • It mostly works (otherwise noone would care).
  • It is written in a way that is hard to test.

Why test?

More specifically why invest in tests that can be executed unattended?

  • Business Value:
    • Shorter development cycles.
    • Faster development of new features.
    • Less bugs reaching the customers.
  • Avoid regression.
  • Better Software Design (This is a side-effect of TDD.)
  • Your Sanity.
    • When your software is well tested you know that the chances of having unexpected downtime are much lower then with a software that was not tested well.

Testing Modes

There are many categories of testing. Let’s list some of them:

  • Functional testing. (Checking the functionality of the software.)
    • Unit testing
    • Integration testing
    • Acceptance testing (BDD Behavior-driven development is related to this.)
    • White box
    • Black box
    • Regression testing
  • Code quality tests
  • Usability testing
  • Performance testing
  • Load testing
  • Stress test
  • Endurance test
  • Security testing
  • …

Other names

  • Smoke testing
  • Sanity testing (preliminary testing).

Testing Applications

  • Web site
  • Web application
  • Web API / Microservice (JSON, XML)
  • Mobile Application
  • Desktop Application (GUI)
  • Command-line tool (CLI)
  • Batch process

Testing: What to test?

How would you check that the code works as expected?

In general we can divide the testing into two:

Happy path

  • Valid input
  • Valid edge cases (0, -1, empty string, etc.)
  • e.g. if you are expecting a number what if you get 0, -1, 131314134141?

Sad path

  • Invalid input?

  • If you are expecting a number what if you get the word “zero”?

  • If you are expecting a number between 0-255 and you get 256? or -1?

  • Broken input (string instead of number, invalid values, too long input, etc.)

  • System failure (power failure, network outage, lack of memory, lack of disk, …)

  • Third-party error or failure - How does your system work if the 3rd party API does not respond or returns rubbish?

Also consider non-functional problems:

  • Extreme load

Testing in Python

  • Doctest
  • Unittest
  • Pytest
  • Nose
  • Nimoy
  • Hypothesis
  • Selenium
  • Tox

Testing Environment

  • Git (or other VCS)
  • Virtualenv
  • Docker
  • …

Testing Setup - Fixture

What kind of environment do you need before you can start testing your code?

  • Web server.
  • Databases.
  • Other machines.
  • Devices.
  • External services.

Testing Resources

The plan of this course

  1. Learn how to write unit-tests for simple functions.
  2. Deal with special problems.
    • How to deal with randomness?
    • How to deal with time?
    • How to deal with reported bugs that we cannot fix now?
    • How to deal with setup and tear-down of the environments?
  3. Then get see what to do if we already have a working application that is hard to test.

The pieces of your software?

  • Web application with HTML interface?
  • Web application with HTML + JavaScript? Which frameworks?
  • Web application with JSON interface? (API)
  • What kind of databases do you have in the system? SQL? NoSQL? What size is the database?
  • Source and the format of your input? Zip? CSV? XML? SQL Dump? JSON?
  • The format of your output? HTML/PDF/CSV/JSON/XML/Excel/Email/..?
  • Are you pushing out your results or are your clients pulling them? e.g. via an API?
  • What external dependencies do you have? Slack, Twilio, What kind of APIs?

Continuous Integration (CI)

There are various service (e.g. GitHub Actions, GitLab Piplines) that allow you (the administrator of a project) to run arbitrary code every time someone pushes out changes to GitHub or GitLab (Etc.) You can configure various checks there, for example you can run all the tests of the project or you can run linters on the project.

  • Reduce feedback cycle.
  • Avoid regression.
  • On every push.
  • It can run every few hours full test coverage.

pre-commit

In each git repository there is a folder called .git/hooks/ where we can place scripts that will be executed at various events. A new git repository will include a examples for each event.

One of the events is called pre-commit. It triggers right before a commit is saved. If it there is a file called .git/hooks/pre-commit, then it is executed automatially by git when the user tries to commit a change.

It can be configured to run various checks and make sure each check passes before the commit can go through.

For example one can configure to run all the tests before each commit or to check if the code is formatted cas expected or if the commit message contains a reference to an open issue. If any one of these checks fails then the commit does not go through.

There is a framework, cleverly also named pre-commit, that help manage the various checks you can configure in the pre-commit hook. It happens to be written in Python, but it can be use with any git repository. Regardless of the content of the git repository.

One caveat, though. Pre-commits are configured at the sole discretion of the person commiting the change. You cannot enforce them on the git server. So as an administrator of a project you will still want to setup a CI system (e.g. GitHub Actions) on the git server. The pre-commit checks are only there to avoid emberassment that you committed a broken change.

AUT - Application Under Test

Before we get started testing, let’s take a look at a couple of very simple applications that we will want to test.

The mymath module

We have a module called mymath that has two methods: add and div. It is a very simple module with very simple functions, but for our purposes that does not matter.

Testing a function should be calling the function with some parameters and then comparing the result to some expected value.

The complexity of the function does not matter.

In this case each function also comes with some documentation, including examples using the interactive shell.


def add(x, y):
    """Adding two numbers

    >>> add(2, 3)
    5

    """
    return x + y

def div(x, y):
    """Dividing two numbers

    >>> div(8, 2)
    4.0
    >>> div(8, 0)
    Traceback (most recent call last):
    ...
    ZeroDivisionError: division by zero
    """
    return x / y

Fibonacci module

This is another simple module, but this one has a bug. Later we’ll discuss much more complex cases, but for the understanding of the Pytest testing framework this simple one will do.

It also has some documentation with a few working examples. We’ll see what happens when someone reports a bug in this. We’ll see how are we going to test it.

def fib(n):
    """
    Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, ...

    >>> fib(7)
    8

    >>> fib(10)
    34

    >>> fib(42)
    165580141
    """
    fibs = [0, 1]
    for _ in range(n-2):
        fibs.append(fibs[-1] + fibs[-2])
    return fibs[-1]

Use the mymath module

Before we start testing it, let’s see how we would use this module?

import mymath
print( mymath.add(2, 3) )
print( mymath.div(6, 2) )

The output looks like this

$ python src/examples/testing/good/use_mymath.py

5
3.0

Some people will write such examples during development, look at the results and conclude that the functions work fine. Unfortunately they will leave these helper programs scattered around their computer and then these would get lost.

Manually test the mymath module

We can create a command line application using the module

import mymath
import sys

if len(sys.argv) != 4:
    exit("Usage: {} [add|div] INT INT".format(sys.argv[0]))

if sys.argv[1] == 'add':
    print(mymath.add(int(sys.argv[2]), int(sys.argv[3])))
if sys.argv[1] == 'div':
    print(mymath.div(int(sys.argv[2]), int(sys.argv[3])))

And then we can ask our QA team to try it:

$ python src/examples/testing/good/run_mymath.py add 19 23
42
$ python src/examples/testing/good/run_mymath.py div 19 23
0.8260869565217391

Use the Fibonacci module

import fibonacci

print(fibonacci.fib(20))

Testing with doctest

doctest allows us to verify that the implementation of our functions and the example we provide in their documentation are aligned.

For every Python function we can add an string as the first thing in the body of the function. (We usually use tripple quotes for this as we are including several lines. In this documentation we can include both free text and examples that look as if we were using the interactive shell.

That is we’ll include a prompt >>>, after the prompt we’ll write a call to the function. Underneath we’ll show the values the function is supposed to return.

Doctest will read our source code. It will read our functions and their documentation.

It will extract the examples from the documentation. Run the function as shown after the >>> prompt and compare the results to what can be found next.

This is sometimes referred to as “literate testing” or “executable documentation”.

Doctest for good module

python -m doctest src/examples/testing/good/mymath.py

In case of success there is no output.

However we can check the exit code which is 0 on success and some other number on failure.

Exit code on Linux and macOS:

$ echo $?
0

Exit code on Windows:

> echo %ERRORLEVEL%
0

We can also use the verbose mode to see the progress:

python -m doctest -v src/examples/testing/good/mymath.py

Doctest for bad module

In this case we have a function that works for some input and returns incorrect results for other input.

It can still have documentation with working examples:

def fib(n):
    """
    Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, ...

    >>> fib(7)
    8

    >>> fib(10)
    34

    >>> fib(42)
    165580141
    """
    fibs = [0, 1]
    for _ in range(n-2):
        fibs.append(fibs[-1] + fibs[-2])
    return fibs[-1]

We can run the tests and they will pass.

python -m doctest src/examples/testing/bad/fibonacci.py

Exit code on Linux and macOS:

$ echo $?
0

Exit code on Windows:

> echo %ERRORLEVEL%
0

What if someone reports the bug? What shall we do?

Doctest for bad module with failure

We can add another test case that was reported to return bad results.

def fib(n):
    """
    Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, ...

    >>> fib(1)
    0

    >>> fib(7)
    8

    >>> fib(10)
    34

    >>> fib(42)
    165580141
    """
    fibs = [0, 1]
    for _ in range(n-2):
        fibs.append(fibs[-1] + fibs[-2])
    return fibs[-1]

We can run doctest:

$ python3 -m doctest fibonacci.py
**********************************************************************
File "src/examples/testing/failure/fibonacci.py", line 5, in fibonacci.fib
Failed example:
    fib(1)
Expected:
    0
Got:
    1
**********************************************************************
1 item had failures:
   1 of   4 in fibonacci.fib
***Test Failed*** 1 failure.

It still checked all the 4 test cases and reported that one of them failed.

The exit code will also indicate the failure.

Exit code on Linux and macOS:

$ echo $?
1

Exit code on Windows:

> echo %ERRORLEVEL%
1

At this point we probably need to fix our code, but for this course we are more interested in the various ways we can test code and how failures are reported.

Testing with unittest

Python comes with the unittest library. It is a simple testing framework and despite its name it can be used for any form of functional tests. Not just unittest.

It is also not very popular any more, but you might encounter it in various places, so it is a good idea to a take a quick look at it.

Unittests are real code, not just documentation. They are usually stored in separate files.

The names of these files usually start with the word test_.

Test the mymath module

Unittests are usually written in separate files.

We need to create one or more classes called TestSomething and inherit from unittest.TestCase.

Inside the class there have to be one or more test-methods. Each one must start with test_.

import unittest
import mymath

class TestMath(unittest.TestCase):

    def test_match(self):
        self.assertEqual(mymath.add(2, 3), 5)
        self.assertEqual(mymath.div(6, 3), 2)
        self.assertEqual(mymath.div(42, 1), 42)
        self.assertEqual(mymath.add(-1, 1), 0)

if __name__ == '__main__':
    unittest.main()

$ python test_mymath_with_unittest.py
.
-------------------------------------------------
Ran 1 test in 0.000s

OK

The single . in this output indicates that we had a single test function and it was successful.

Alternatively we can leave out the last two lines:

if __name__ == '__main__':
    unittest.main()

And use the unittest itself to run the tests.

python -m unittest test_mymath_with_unittest.py

Each test case in its own method

We can put each check in a separate function.

import unittest
import mymath

class TestMath(unittest.TestCase):

    def test_add_2_3(self):
        self.assertEqual(mymath.add(2, 3), 5)

    def test_div_6_3(self):
        self.assertEqual(mymath.div(6, 3), 2)

    def test_div_42_1(self):
        self.assertEqual(mymath.div(42, 1), 42)

    def test_add_1_1(self):
        self.assertEqual(mymath.add(-1, 1), 0)

#if __name__ == '__main__':
#    unittest.main()

In this case the reporting will show 4 tests.

$ python -m unittest test_mymath_with_unittest_separated.py 
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

The advantage of putting them in separate functions is that then the tests can run independently.

Test the Fibonacci

import unittest
from fibonacci import fib

class TestFibo(unittest.TestCase):

    def test_fib(self):
        self.assertEqual(fib(7), 8)
        self.assertEqual(fib(10), 34)
        self.assertEqual(fib(42), 165580141)

We don’t need the extra code to trigger the unittest from inside our module. We can just run it using the follow command line:

$ python -m unittest test_fibonacci_with_unittest.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Test the Fibonacci failure

import unittest
from fibonacci import fib

class TestFibo(unittest.TestCase):

    def test_fib(self):
        self.assertEqual(fib(1), 0)
        self.assertEqual(fib(7), 8)
        self.assertEqual(fib(10), 34)
        self.assertEqual(fib(42), 165580141)


$ python -m unittest test_fibonacci_with_unittest_failure.py
F
======================================================================
FAIL: test_fib (test_fibonacci_with_unittest_failure.TestFibo.test_fib)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "src/examples/testing/bad/test_fibonacci_with_unittest_failure.py", line 7, in test_fib
    self.assertEqual(fib(1), 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

$ echo $?
1

We can see that there is a problem, we can even see the error message, but we don’t know what would be the result of the other 3 test-cases. Because the first test-case already raised an exception and stopped the processing, we don’t know if the other would pass or if all the others would fail as well.

Maybe in our current situation it is not that important. After all we just added a test-case and we know the others passed before we added this. However if you have a bunch of tests. You make a change to your code and then you see such a failure then you’d want to know if the change impacted only one test-case, more than one case or all of them.

Test the Fibonacci separated

Just as in the previous case, we can separate the test-cases into separate functions. Soon we’ll see the real value of this separation.

import unittest
from fibonacci import fib

class TestFibo(unittest.TestCase):

    def test_7(self):
        self.assertEqual(fib(7), 8)

    def test_10(self):
        self.assertEqual(fib(10), 34)

    def test_42(self):
        self.assertEqual(fib(42), 165580141)


$ python -m unittest test_fibonacci_with_unittest_separated.py 
...
-------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Test the Fibonacci failure separated

Now if we add a 4th test function and run the test we can see it reports F.... F meaning the first test-case failed and the 3 dots indicating that the other 3 test-cases passed.

import unittest
from fibonacci import fib

class TestFibo(unittest.TestCase):

    def test_1(self):
        self.assertEqual(fib(1), 0)

    def test_7(self):
        self.assertEqual(fib(7), 8)

    def test_10(self):
        self.assertEqual(fib(10), 34)

    def test_42(self):
        self.assertEqual(fib(42), 165580141)


$ python -m unittest test_fibonacci_with_unittest_failure_separated.py 
F...
======================================================================
FAIL: test_1 (test_fibonacci_with_unittest_failure_separated.TestFibo.test_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gabor/github/code-maven.com/python.code-maven.com/books/python-testing/src/examples/testing/bad/test_fibonacci_with_unittest_failure_separated.py", line 7, in test_1
    self.assertEqual(fib(1), 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=1)

Testing - skeleton

import unittest

def add(x, y):
    return x+y

class Something(unittest.TestCase):

    def setUp(self):
        pass
        #print("setup")

    def tearDown(self):
        pass
        #print("teardown")

    def test_something(self):
        self.assertEqual(add(2, 3), 5)
        self.assertEqual(add(0, 3), 3)
        self.assertEqual(add(0, 3), 2)


    def test_other(self):
        self.assertEqual(add(-3, 3), 0)
        self.assertEqual(add(-3, 2), 7)
        self.assertEqual(add(-3, 2), 0)


if __name__ == '__main__':
    unittest.main()

Testing

import unittest

class TestReg(unittest.TestCase):

    def setUp(self):
        self.str_number = "123"
        self.str_not_number = "12x"

    def test_match1(self):
        self.assertEqual(1, 1)
        self.assertRegexpMatches(self.str_number, r'^\d+$')

    def test_match2(self):
        self.assertEqual(1, 1)
        self.assertRegexpMatches(self.str_not_number, r'^\d+$')

if __name__ == '__main__':
    unittest.main()

Test examples

Testing with PyTest

pytest is the de-facto standard library for testing Python code.

Pytest features

  • Organize and run tests per directory (test discovery).
  • Run tests by name matching.
  • Run tests by mark (smoke, integration, db).
  • Run tests in parallel with the xdist plugin.
  • A wealth of plugins. (e.g. test coverage reporting.)
  • Create your own fixtures and distribute them.
  • Create your own plugins and distribute them.

Pytest setup

You will need to install pytest before you can use it. It should be added as a development-time dependency of the project.

Python 2

virtualenv venv
source venv/bin/activate
pip install pytest

Python 3

virtualenv venv -p python3
source venv/bin/activate
pip install pytest

Python 3 Debian/Ubuntu

apt-get install python3-pytest

Python 3 RedHat/Centos

yum install python3-pytest

Testing functions

As we start writing tests to be used by pytest we don’t actually need to use pytest in the code. We just need to create one or more files with a name starting with test_.

In those files one or more “test-functions”, that is functions that start with test_.

Inside those function we can call the functions of the AUT and compare the returned values to the expected values passing the result to the regular assert function of Python.

The assert function receives a boolean value. If it is True then assert does nothing. If it is False then assert raises an exception.

We should not call these test-functions ourselves! Pytest will do it for us.

import mymath

def test_math():
    assert mymath.add(2, 3)  == 5
    assert mymath.div(6, 3)  == 2
    assert mymath.div(42, 1) == 42
    assert mymath.add(-1, 1) == 0

Running the tests:

The place where we do need pytest is for running the tests.

pytest test_mymath_with_pytest.py

============================= test session starts ==============================
platform darwin -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/gabor/work/training/python, inifile:
collected 1 item

examples/pytest/test_mymath.py .                                         [100%]

=========================== 1 passed in 0.01 seconds ===========================

Pytest will read the content of the file(s), find the test_... functions and run each one of them. If any of them raises an exception (because of the assert or otherwise`), pytest will collect this data and create a report at the end.

In the above report you can see that pytest reported the name of the test-file. The single dot . following the filename indicates that there was one test function that passed.

After the test run we could also see the exit code of the program by typing in echo $? on Linux or Mac or echo %ERRORLEVEL% on Windows.

$ echo $?
0

> echo %ERRORLEVEL%
0

Testing class and methods

Pytest allows us to write our tests in an object oriented fashion as well. In that case we need to create one or more classes called ClassSomething in the files that are called test_something.py. In the classes there have to be methods called test_something that are similar to the function in the previous case.

import mymath

class TestMath():
    def test_math(self):
        assert mymath.add(2, 3)  == 5
        assert mymath.div(6, 3)  == 2
        assert mymath.div(42, 1) == 42
        assert mymath.add(-1, 1) == 0

We run the tests the same way.

$ pytest test_mymath_with_pytest_class.py
=========================================== test session starts ============================================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 1 item

test_mymath_with_pytest_class.py .                                                                   [100%]

============================================ 1 passed in 0.07s =============================================

Shall we use OOP for writing tests?

Test are supposed to be a lot less complex than our real code. Therefore in general I discourage using OOP in the tests.

Pytest - passing test for the Fibonacci function

from fibonacci import fib


def test_fib():
    assert fib(7)  == 8
    assert fib(10) == 34
    assert fib(42) == 165580141

We can run the tests in two different ways. The regular would be to type in pytest and the name of the test file. In some setups this might not work and then we can also run python -m pytest and the name of the test file.

pytest test_fibonacci_with_pytest.py
python -m pytest test_fibonacci_with_pytest.py

============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 1 item

test_fibonacci_with_pytest.py .                                          [100%]

============================== 1 passed in 0.05s ===============================

The top of the output shows some information about the environment, (version numbers, plugins) then “collected” tells us how many test-cases were found by pytest. Each test function is one test case.

Then we see the name of the test file and a single dot indicating that there was one test-case and it was successful.

After the test run we could also see the exit code of the program by typing in echo $? on Linux or Mac or echo %ERRORLEVEL% on Windows.

$ echo $?
0

> echo %ERRORLEVEL%
0

Pytest failing test in one function

Once we had that passing test we might have shared our code just to receive complaints that it does not always work properly. Specifically passing in 1 should return , but it returns 1.

So for your investigation the first thing you need to do is to write a test case expecting it to work proving that your code works. So you add a second assertion.

from fibonacci import fib


def test_fib():
    assert fib(1)  == 0
    assert fib(7)  == 8
    assert fib(10) == 34
    assert fib(42) == 165580141

To your surprise the tests fails with the following output:

$ pytest test_fibonacci_with_pytest_failing.py
=========================================== test session starts ============================================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 1 item

test_fibonacci_with_pytest_failing.py F                                                              [100%]

================================================= FAILURES =================================================
_________________________________________________ test_fib _________________________________________________

    def test_fib():
>       assert fib(1)  == 0
E       assert 1 == 0
E        +  where 1 = fib(1)

test_fibonacci_with_pytest_failing.py:5: AssertionError
========================================= short test summary info ==========================================
FAILED test_fibonacci_with_pytest_failing.py::test_fib - assert 1 == 0
============================================ 1 failed in 0.12s =============================================

We see the collected 1 item because we still only have one test function.

Then next to the test file we see the letter F indicating that we had a single test failure.

Then we can see the details of the test failure. Among other things we can see the actual value returned by the fib function and the expected value.

Knowing that assert only receives the True or False values of the comparison, you might wonder how did this happen. This is part of the magic of pytest. It uses some introspection to see what was in the expression that was passed to assert and it can print out the details helping us see what was the expected value and what was the actual value. This can help understanding the real problem behind the scenes.

You can also check the exit code and it will be something different from 0 indicating that something did not work. The exit code is used by CI-systems to see which test run were successful and which failed.

$ echo $?
1

> echo %ERRORLEVEL%
1

One big disadvantage of having two or more asserts in the same test function is that we don’t know what would be the result of the other asserts.

Pytest failing test separated

Instead of putting the several asserts in the same test function we could also put them in separate ones like in this example.

from fibonacci import fib


def test_fib_1():
    assert fib(1)  == 0

def test_fib_7():
    assert fib(7)  == 8

def test_fib_10():
    assert fib(10) == 34

def test_fib_42():
    assert fib(42) == 165580141

The result of running this test file shows that it collected 4 items as there were 4 test functions.

Then next to the test file we see an F indicating the failed test and 3 dot indicating the successful test cases. The more detailed test report helps.

At the bottom of the report you can also see that now it indicates 1 failed and 3 passed test.

$ pytest test_fibonacci_with_pytest_failing_separated.py
=========================================== test session starts ============================================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 4 items

test_fibonacci_with_pytest_failing_separated.py F...                                                 [100%]

================================================= FAILURES =================================================
________________________________________________ test_fib_1 ________________________________________________

    def test_fib_1():
>       assert fib(1)  == 0
E       assert 1 == 0
E        +  where 1 = fib(1)

test_fibonacci_with_pytest_failing_separated.py:5: AssertionError
========================================= short test summary info ==========================================
FAILED test_fibonacci_with_pytest_failing_separated.py::test_fib_1 - assert 1 == 0
======================================= 1 failed, 3 passed in 0.09s ========================================

Writing those separate test functions is annoying and creates too much repetition. We’ll fix that later.

For now we should just appreciate the simplicity of the test code.

Again you might feel the urge to fix the bug, but our focus is writing tests. Besides, you might be busy working on a feature or on another bug.

Pytest run all the test files

In the src/examples/testing/good folder run pytest.

$ pytest
=========================================== test session starts ============================================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 7 items

test_mymath_with_pytest.py .                                                                         [ 14%]
test_mymath_with_pytest_class.py .                                                                   [ 28%]
test_mymath_with_unittest.py .                                                                       [ 42%]
test_mymath_with_unittest_separated.py ....                                                          [100%]

============================================ 7 passed in 0.06s =============================================

It found all the test_* files and all the test_* functions. (and also the Test* class and the test_ methods). It ran them all and reported the results.

If you run the same in the src/examples/testing/ folder then it will find a lot more test-cases, many of which will fail as we had a number of examples showing failure.

Exercise: test simple module

Take the standard math library and write tests for some of the functions. Actually take any standard Python module or functionality and write a few tests.

If you find a bug, report it here.

Solution: test simple module

import math

def test_comb():
    assert math.comb(10, 3) == 120;


def test_factorial():
    assert math.factorial(10) == 3628800;

def test_gcd():
    assert math.gcd(100, 60, 130) == 10;

pytest test_math.py

Pytest basic command line options

Getting help for pytest -h

pytest -h

Pytest verbose mode -v

In the verbose mode pytest will show the name and the status of each test.

$ pytest -v test_mymod_1.py

test_mymod_1.py::test_anagram PASSED

$ pytest -v test_mymod_2.py

test_mymod_2.py::test_anagram PASSED
test_mymod_2.py::test_multiword_anagram FAILED

Pytest quiet mode

  • -q
$ pytest -q test_mymod_1.py
.
1 passed in 0.01 seconds

$ pytest -q test_mymod_2.py

.F
=========================== FAILURES ===========================
____________________ test_multiword_anagram ____________________

    def test_multiword_anagram():
       assert is_anagram("ana gram", "naga ram")
>      assert is_anagram("anagram", "nag a ram")
E      AssertionError: assert False
E       +  where False = is_anagram('anagram', 'nag a ram')

test_mymod_2.py:10: AssertionError
1 failed, 1 passed in 0.09 seconds

PyTest print STDOUT and STDERR using -s

import sys
import appprint

def test_hello():
    print("hello testing")
    print("stderr during testing", file=sys.stderr)

def test_add():
    assert appprint.add(2, 3) == 5

Using also -q makes it easier to see the printed text.

$ pytest -s -q test_stdout_stderr.py
hello testing
stderr during testing
.Adding 2 to 3
.
2 passed in 0.01 seconds

Pytest: show extra test summary info with -r

Soon we’ll learn about the ability to mark certain test functions as expected to fail and others to be skipped in some condition. When running these tests we’ll want to be able list these based on their special feature.

Using the -v flag would just list all of them which, if you have a lot of tests, would be messy.

The letters one can use with the -r flag:

  • (f)ailed
  • (E)error
  • (s)skipped
  • (x)failed
  • (X)passed
  • (p)passed
  • (P)passed with output
  • (a)all except pP
pytest -rx  - xfail, expected to fail
pytest -rs  - skipped
pytest -ra  - all the special cases

import pytest

def test_pass():
    assert True

def test_fail():
    assert False

@pytest.mark.skip(reason="Unconditional skip")
def test_with_skip():
    assert True

@pytest.mark.skipif(True, reason="Conditional skip")
def test_with_skipif():
    assert True

@pytest.mark.skipif(False, reason="Conditional skip")
def test_with_skipif_but_run():
    assert True


@pytest.mark.xfail(reason = "Expect to fail and failed")
def test_with_xfail_and_fail():
   assert False

@pytest.mark.xfail(reason = "Expect to fail but passed")
def test_with_xfail_but_pass():
   assert True

Pytest expected exception

  • What if raising an exception is part of the specification of a function?
  • That given certain (incorrect) input it will raise a certain exception?
  • How can we test that we get the right exception. The expected exception?

Pytest a nice Fibonacci example

This is an implementation of the Fibonacci function.

def fib(n):
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a

Pytest testing Fibonacci

We already know how to write a test checking the regular result.

from fibonacci import fib

def test_fib():
    assert fib(10) == 55

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib1
plugins: flake8-1.0.6, dash-1.17.0
collected 1 item

test_fibonacci.py .                                                      [100%]

============================== 1 passed in 0.00s ===============================

Using Fibonacci with a negative number

What if the user calls it with -3 ? We get the result to be 1. We don’t want that.

from fibonacci import fib

print(fib(10))
print(fib(-3))

Output:

55
1

Pytest raise exception

We change our implementation of the Fibonacci function to raise an exception if the user provided a negative number.

def fib(n):
    if n < 1:
        raise ValueError(f'Invalid parameter {n}')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


from fibonacci import fib

print(fib(10))
print(fib(-3))

Output:

55
Traceback (most recent call last):
  File "use_fib.py", line 4, in <module>
    print(fib(-1))
  File "fibonacci.py", line 3, in fib
    raise ValueError(f'Invalid parameter {n}')
ValueError: Invalid parameter -1

Pytest testing expected exception

Pytest provides a tool that allows us to “expect and exception”. We’ll usually use it using a with context manager.

Here we can see two different ways to verify the exception.

import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1'

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib2
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_fibonacci.py ...                                                    [100%]

============================== 3 passed in 0.01s ===============================

Now that we are testing the exception let’s see how will this test protect us?

  • Someone changing the text of the exception.
  • Someone removing the exception.

Pytest Change the text of the exception

What if someone changes the text of the exception?

def fib(n):
    if n < 1:
        raise ValueError(f'Invalid parameter was given {n}')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1'

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib3
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_fibonacci.py .FF                                                    [100%]

=================================== FAILURES ===================================
______________________________ test_fib_negative _______________________________

    def test_fib_negative():
        with pytest.raises(Exception) as err:
            fib(-1)
        assert err.type == ValueError
>       assert str(err.value) == 'Invalid parameter -1'
E       AssertionError: assert 'Invalid para... was given -1' == 'Invalid parameter -1'
E         - Invalid parameter -1
E         + Invalid parameter was given -1
E         ?                   ++++++++++

test_fibonacci.py:11: AssertionError
___________________________ test_fib_negative_again ____________________________

    def test_fib_negative_again():
        with pytest.raises(ValueError) as err:
            fib(-1)
>       assert str(err.value) == 'Invalid parameter -1'
E       AssertionError: assert 'Invalid para... was given -1' == 'Invalid parameter -1'
E         - Invalid parameter -1
E         + Invalid parameter was given -1
E         ?                   ++++++++++

test_fibonacci.py:16: AssertionError
=========================== short test summary info ============================
FAILED test_fibonacci.py::test_fib_negative - AssertionError: assert 'Invalid...
FAILED test_fibonacci.py::test_fib_negative_again - AssertionError: assert 'I...
========================= 2 failed, 1 passed in 0.03s ==========================

If the change was unintentional then the test helped us catch this. If it was intentional then someone will have to adjust the test as well.

Pytest Missing exception

What if someone comments out or removes the code raising the exception?

def fib(n):
#    if n < 1:
#        raise ValueError(f'Invalid parameter {n}')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1'

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib4
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_fibonacci.py .FF                                                    [100%]

=================================== FAILURES ===================================
______________________________ test_fib_negative _______________________________

    def test_fib_negative():
        with pytest.raises(Exception) as err:
>           fib(-1)
E           Failed: DID NOT RAISE <class 'Exception'>

test_fibonacci.py:9: Failed
___________________________ test_fib_negative_again ____________________________

    def test_fib_negative_again():
        with pytest.raises(ValueError) as err:
>           fib(-1)
E           Failed: DID NOT RAISE <class 'ValueError'>

test_fibonacci.py:15: Failed
=========================== short test summary info ============================
FAILED test_fibonacci.py::test_fib_negative - Failed: DID NOT RAISE <class 'E...
FAILED test_fibonacci.py::test_fib_negative_again - Failed: DID NOT RAISE <cl...
========================= 2 failed, 1 passed in 0.03s ==========================

Pytest Other exception is raised

What if someone changes the type of the exception?

def fib(n):
    if n < 1:
        raise Exception(f'Invalid parameter {n}')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1'

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib5
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_fibonacci.py .FF                                                    [100%]

=================================== FAILURES ===================================
______________________________ test_fib_negative _______________________________

    def test_fib_negative():
        with pytest.raises(Exception) as err:
            fib(-1)
>       assert err.type == ValueError
E       AssertionError: assert <class 'Exception'> == ValueError
E        +  where <class 'Exception'> = <ExceptionInfo Exception('Invalid parameter -1') tblen=2>.type

test_fibonacci.py:10: AssertionError
___________________________ test_fib_negative_again ____________________________

    def test_fib_negative_again():
        with pytest.raises(ValueError) as err:
>           fib(-1)

test_fibonacci.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

n = -1

    def fib(n):
        if n < 1:
>           raise Exception(f'Invalid parameter {n}')
E           Exception: Invalid parameter -1

fibonacci.py:3: Exception
=========================== short test summary info ============================
FAILED test_fibonacci.py::test_fib_negative - AssertionError: assert <class '...
FAILED test_fibonacci.py::test_fib_negative_again - Exception: Invalid parame...
========================= 2 failed, 1 passed in 0.03s ==========================

Pytest No exception is raised

What if instead of raising an exception someone decides to return None?

def fib(n):
    if n < 1:
        return None
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1'

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/fib6
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_fibonacci.py .FF                                                    [100%]

=================================== FAILURES ===================================
______________________________ test_fib_negative _______________________________

    def test_fib_negative():
        with pytest.raises(Exception) as err:
>           fib(-1)
E           Failed: DID NOT RAISE <class 'Exception'>

test_fibonacci.py:9: Failed
___________________________ test_fib_negative_again ____________________________

    def test_fib_negative_again():
        with pytest.raises(ValueError) as err:
>           fib(-1)
E           Failed: DID NOT RAISE <class 'ValueError'>

test_fibonacci.py:15: Failed
=========================== short test summary info ============================
FAILED test_fibonacci.py::test_fib_negative - Failed: DID NOT RAISE <class 'E...
FAILED test_fibonacci.py::test_fib_negative_again - Failed: DID NOT RAISE <cl...
========================= 2 failed, 1 passed in 0.03s ==========================

Exercise: test more exceptions

  • Find another case that will break the code.
  • Then make changes to the code that it will no break.
  • The write a test to verify it.

Solution: test more exceptions

def fib(n):
    if int(n) != n:
        raise ValueError(f'Invalid parameter {n} is not an integer')
    if n < 1:
        raise ValueError(f'Invalid parameter {n} is negative')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1 is negative'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1 is negative'

def test_fib_floating_point():
    with pytest.raises(ValueError) as err:
        fib(3.5)
    assert str(err.value) == 'Invalid parameter 3.5 is not an integer'

def test_fib_small_floating_point():
    with pytest.raises(ValueError) as err:
        fib(0.5)
    assert str(err.value) == 'Invalid parameter 0.5 is not an integer'

Output:

============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 5 items

test_fibonacci.py .....                                                  [100%]

============================== 5 passed in 0.07s ===============================

Another Solution: test more exceptions

def fib(n):
    var_type = type(n).__name__
    if var_type != 'int':
        raise ValueError(f'Invalid value type of "{n}" is "{var_type}"')
    if n < 1:
        raise ValueError(f'Invalid parameter {n} is negative')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a


import pytest
from fibonacci import fib

def test_fib():
    assert fib(10) == 55

def test_fib_negative():
    with pytest.raises(Exception) as err:
        fib(-1)
    assert err.type == ValueError
    assert str(err.value) == 'Invalid parameter -1 is negative'

def test_fib_negative_again():
    with pytest.raises(ValueError) as err:
        fib(-1)
    assert str(err.value) == 'Invalid parameter -1 is negative'

def test_fib_floating_point():
    with pytest.raises(ValueError) as err:
        fib(3.5)
    assert str(err.value) == 'Invalid value type of "3.5" is "float"'

def test_fib_small_floating_point():
    with pytest.raises(ValueError) as err:
        fib(0.5)
    assert str(err.value) == 'Invalid value type of "0.5" is "float"'

def test_fib_big_floating_point():
    with pytest.raises(ValueError) as err:
        fib(3.0)
    assert str(err.value) == 'Invalid value type of "3.0" is "float"'

Output:

============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/gabor/github/code-maven.com/python.code-maven.com
configfile: pyproject.toml
plugins: anyio-4.12.0, cov-7.0.0
collected 6 items

test_fibonacci.py ......                                                 [100%]

============================== 6 passed in 0.04s ===============================

Exercise: test exceptions in the math functions

Some of the function in the standard math library will raise exceptions for certain input.

Find a few and write tests for them.

Solution: test exceptions in the math functions

import math
import pytest

def test_math():
    with pytest.raises(Exception) as exinfo:
        math.factorial(-1)
    assert exinfo.type == ValueError
    assert str(exinfo.value) == 'factorial() not defined for negative values'

    with pytest.raises(Exception) as exinfo:
        math.factorial(1.2)
    assert exinfo.type == TypeError
    assert str(exinfo.value) == "'float' object cannot be interpreted as an integer"

    # Actually, if I am not mistaken this used to be a ValueError with this message:
    # 'factorial() only accepts integral values'

Handling failing tests (bugs)

Let’s assume you already have some code with some tests and all the tests are passing.

You encounter a bug or someone reports a bug.

The first thing you probably want to do is to reproduce the bug locally. Without being able to reproduce a bug it is usually very hard to fix it.

Once your reproduced the bug you will probably want to convert that into a test case.

Actually if you already have a good testing environment, it might be easier to just write the test that fails due to the bug.

Now what?

If you have the time to fix the bug then you’ll probably do that, but what if you are in the middle of something and you know fixing this bug will take a lot of time? What should you do with the failing test?

You can’t just add it to the code-base as that will mean your CI starts failing and people start asking question.

You don’t want to hide the test either.

You can mark is a test that is expected to fail. (in other programming languages it is called a TODO-test).

Your test will still run and it will still fail, but pytest will report it as expected to fail and it will not break your CI.

Then when you have time to fix this bug, you already have the test and you just need to remove the xfail mark.

Also if you or someone else accidently fixes this bug, you’ll notice.

Pytest simple module to be tested

An anagram is a pair of words containing the exact same letters in different order. For example:

  • listen silent
  • elvis lives
def is_anagram(a_word, b_word):
    return sorted(a_word) == sorted(b_word)

#return sorted(filter(lambda v: v != " ", a_word)) == sorted(filter(lambda v: v != " ", b_word))

Pytest simple tests - success

from anagram import is_anagram

def test_anagram():
    assert is_anagram("elvis", "lives")
    assert is_anagram("silent", "listen")
    assert not is_anagram("one", "two")

Pytest simple tests - success output

$ pytest test_mymod_1.py

===================== test session starts ======================
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /examples/python/pt, inifile:
collected 1 items

test_mymod_1.py .

=================== 1 passed in 0.03 seconds ===================

Pytest simple tests - failure

  • Failure reported by user: is_anagram(“anagram”, “nag a ram”) is expected to return true.
  • We write a test case to reproduce the problem. It should fail now.
from anagram import is_anagram

def test_anagram():
    assert is_anagram("elvis", "lives")
    assert is_anagram("silent", "listen")
    assert not is_anagram("one", "two")

def test_multiword_anagram():
    assert is_anagram("ana gram", "naga ram")
    assert is_anagram("anagram", "nag a ram")

Pytest simple tests - failure output

$ pytest test_mymod_2.py

===================== test session starts ======================
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /examples/python/pt, inifile:
collected 2 items

test_mymod_2.py .F

=========================== FAILURES ===========================
____________________ test_multiword_anagram ____________________

    def test_multiword_anagram():
       assert is_anagram("ana gram", "naga ram")
>      assert is_anagram("anagram", "nag a ram")
E      AssertionError: assert False
E       +  where False = is_anagram('anagram', 'nag a ram')

test_mymod_2.py:10: AssertionError
============== 1 failed, 1 passed in 0.09 seconds ==============

Pytest: expect a test to fail (xfail or TODO tests)

Use the @pytest.mark.xfail decorator to mark the test.

from anagram import is_anagram
#from fixed_anagram import is_anagram
import pytest

def test_anagram():
   assert is_anagram("abc", "acb")
   assert is_anagram("silent", "listen")
   assert not is_anagram("one", "two")

@pytest.mark.xfail(reason = "Bug #42")
def test_multiword_anagram():
   assert is_anagram("ana gram", "naga ram")
   assert is_anagram("anagram", "nag a ram")

Pytest: expect a test to fail (xfail or TODO tests)

$ pytest test_mymod_3.py

======= test session starts =======
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
Using --random-order-bucket=module
Using --random-order-seed=557111

rootdir: /Users/gabor/work/training/python/examples/pytest, inifile:
plugins: xdist-1.16.0, random-order-0.5.4
collected 2 items

test_mymod_3.py .x

===== 1 passed, 1 xfailed in 0.08 seconds =====

PyTest: show xfailed tests with -rx

$ pytest -rx test_mymod_3.py

======= test session starts =======
platform darwin -- Python 3.5.2, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
Using --random-order-bucket=module
Using --random-order-seed=557111

rootdir: /Users/gabor/work/training/python/examples/pytest, inifile:
plugins: xdist-1.16.0, random-order-0.5.4
collected 2 items

test_mymod_3.py .x

===== short test summary info =====
XFAIL test_mymod_3.py::test_multiword_anagram
  Bug #42

===== 1 passed, 1 xfailed in 0.08 seconds =====

Fixed Anagram

def is_anagram(a_word, b_word):
    return sorted(a_word.replace(' ', '')) == sorted(b_word.replace(' ', ''))

Show when xfail passes.

Multiple test functions - selection

  • What if there are more than one test functions?
  • What if some of them fail?
  • How can I stop at the first failure?
  • How can I run a specific test to focus on fixing that?

Multiple Failures in separate test functions

Every test function is executed separately. Normally pytest will run all the test functions, collect the results and display a summary. In this example we have 5 test functions. We don’t really test anything in them but calling assert False will fail the given test.

def test_one():
    print('one before')
    assert True
    print('one after')

def test_two():
    print('two before')
    assert False
    print('two after')

def test_three():
    print('three before')
    assert True
    print('three after')

def test_four():
    print('four before')
    assert False
    print('four after')

def test_five():
    print('five before')
    assert True
    print('five after')

We can run this as pytest test_failures.py and observe the results.

PyTest: Multiple Failures output

The normal output will indicate that 3 of the 5 tests passed and 2 failed:

test_failures.py .F.F.

We can use the verbose flag (-v) to get a more detailed report:

$ pytest -v test_failures.py

test_failures.py::test_one PASSED
test_failures.py::test_two FAILED
test_failures.py::test_three PASSED
test_failures.py::test_four FAILED
test_failures.py::test_five PASSED

We can also use the -s flag to let pytest show what we printed on the screen during the test run. This will show the text in the cases of the where the assertion was successful we can see the print statements both before and after the assertion. In the cases when the assertion failed we only see the print statements before the assertion.

$ pytest -s test_failures.py

one before
one after
two before
three before
three after
four before
five before
five after

PyTest: stop on first failure -x, --maxfail

Seeing all the failures might help in some case, but in in some other cases, especially during development, you might not want to wait for all the tests to run.

You might want to stop at the first test failure and fix that.

You can achieve that by using the -x flag. In the more generic case you can also use the --maxfail N flag where you can explicitly say to stop running after N failures.

pytest -x
pytest --maxfail 1

PyTest Selective running of test functions

During development you will probably want to focus on a specific test (or a specific group of tests). You can run a specific test by providing its name:

pytest test_failures.py::test_one

pytest test_failures.py::test_two

Using the verbose flag (-v) can help you get the list of tests.

Pytest: skipping tests

There can be various reasons why you might want to skip certain tests.

For example your application might have features and tests checking those features that are Operating system specific. If there is a Windows-spefic feature then there is no point in trying to run in on Linux and macOS.

There can be also features and tests that require special equipment. e.g. On might need a Postgres database and values in environent variables holding the hostname, database name, and the user credential to be used in the test. If those values are not provided then you’ll want to skip the given test.

In those cases you’ll want to skipt the specific test conditionall.

Then there can be tests that have not been fully implemented or that for some reason cannot currently run. Not that they fail, they will crash. In that case you will always want to skip the tests.

Pytest: Operating System specific test

import sys
import pytest

@pytest.mark.skipif(sys.platform != 'darwin', reason="Mac tests")
def test_mac():
    assert True

@pytest.mark.skipif(sys.platform != 'linux', reason="Linux tests")
def test_linux():
    assert True

@pytest.mark.skipif(sys.platform != 'win32', reason="Windows tests")
def test_windows():
    assert True

@pytest.mark.skip(reason="To show we can skip tests without any condition.")
def test_any():
    assert True

pytest test_on_condition.py

collected 4 items

test_on_condition.py ss.s

==== 1 passed, 3 skipped in 0.02 seconds ====

Pytest: show skipped tests with -rs

  • -rs
$ pytest -rs test_on_condition.py

collected 4 items

test_on_condition.py s.ss

===== short test summary info =====
SKIP [1] test_on_condition.py:15: To show we can skip tests without any condition.
SKIP [1] test_on_condition.py:7: Linux tests
SKIP [1] test_on_condition.py:11: Windows tests

==== 1 passed, 3 skipped in 0.03 seconds ====

Pytest: skipping tests output in verbose mode

$ pytest -v test_on_condition.py

test_on_condition.py::test_mac PASSED
test_on_condition.py::test_any SKIPPED
test_on_condition.py::test_windows SKIPPED
test_on_condition.py::test_linux SKIPPED

==== 1 passed, 3 skipped in 0.01 seconds ======

Parametrize tests

There can be a lot of cases when we would like to run the same test-code with different input and different expected output values.

Earlier we created copies of the test functions, but that is not really the nice way to do it.

It requires us to think about a new name for each test function. It means a lot of code repetition.

Let’s see how to eliminate those problems.

Parametrize PyTest with pytest.mark.parametrize testing len

Let’s test the built-in len function.

Test len without parameterization

We can write several test function. One for each string:

import pytest

def test_case_foo():
    assert len("Foo") == 3

def test_case_bar():
    assert len("Bar") == 3

def test_case_emojies():
    assert len("🐍🐪🦀") == 3

Test len with 1 parameter

We can parametrize the function with the input string, but then we’ll need to have a test function for each length.

import pytest

@pytest.mark.parametrize("text", ["Foo", "Bar", "🐍🐪🦀"])
def test_cases_3(text):
    assert len(text) == 3

@pytest.mark.parametrize("text", ["apple", "banan"])
def test_cases_5(text):
    assert len(text) == 5

Test len with 2 parameters

We can use 2 parameters in the parametrization.

import pytest

@pytest.mark.parametrize("text", ["Foo", "Bar", "🐍🐪🦀"])
def test_cases_3(text):
    assert len(text) == 3

@pytest.mark.parametrize("text", ["apple", "banan"])
def test_cases_5(text):
    assert len(text) == 5


@pytest.mark.parametrize("text,length", [
    ("Foo", 3),
    ("Bar", 3),
    ("🐍🐪🦀", 3),
    ("apple", 5),
    ("banana", 6),
    ])
def test_cases(text, length):
    assert len(text) == length

Output:

..........                                                               [100%]
10 passed in 0.07s

mymath

We take a very simple version of the mymath module:

def add(x, y):
    return x + y

One test function

import mymath

# We can have all the cases in one test function
def test_add_all():
    assert mymath.add(1, 1)  == 2
    assert mymath.add(2, 3)  == 5
    assert mymath.add(-1, 1)  == 0

Separate test functions

import mymath

# We can separate them to several test functions
def test_add_1_1():
    assert mymath.add(1, 1)  == 2

def test_add_2_3():
    assert mymath.add(2, 3)  == 5

def test_add_minus_1_1():
    assert mymath.add(-1, 1)  == 0

Test cases of mymath

We could create a data structure holding all the cases and add a loop to the test function. e.g. to test the add function we could create list of tuples where in each tuple the first two values are the input values and the last one is the expected output.

import mymath

# We can create a dataset of the test-cases and loop over them.
def test_add_cases():
    cases = [
        (1, 1, 2),
        (2, 3, 5),
        (-1, 1, 0),
    ]
    for (a, b, expected) in cases:
        assert mymath.add(a, b) == expected

Pytest parameters

import mymath
import pytest

@pytest.mark.parametrize("a,b,expected", [
    (1, 1, 2),
    (2, 3, 5),
    (-1, 1, 0),
    (-1, -1, -2),
])
def test_add(a, b, expected):
    assert mymath.add(a, b)  == expected

Pytest parameters variable

import mymath
import pytest

cases = [
    (1, 1, 2),
    (2, 3, 5),
    (-1, 1, 0),
    (-1, -1, -2),
]

@pytest.mark.parametrize("a,b,expected", cases)
def test_add(a, b, expected):
    assert mymath.add(a, b)  == expected

Pytest parameters external data file

We can move the data of the test-cases into external CSV or Excel file. Let someone els update and maintain the cases.

a,   b, expected
 1,  1,  2
 2,  3,  5
-1,  1,  0
-1, -1, -2

Our code then reads the data from the data-file and uses that to fill the parameters.

import mymath
import pytest
import csv

filename = "cases.csv"

cases = []
with open(filename, "r") as fh:
    reader = csv.reader(fh)
    next(reader, None) # skip first row
    for line in reader:
        cases.append(tuple(map(int, line)))
        #cases.append(list(map(int, line)))
        #cases.append(list(map(float, line)))
print(cases)

@pytest.mark.parametrize("a,b,expected", cases)
def test_add(a, b, expected):
    assert mymath.add(a, b)  == expected

Pramatrization matrix

If we add several separate decorators, then we create a matrix of all the test cases. This for example will run 3x3 = 9 test-cases.

import mymath
import pytest

@pytest.mark.parametrize("a", [1, 2, 3])
@pytest.mark.parametrize("b", [4, 5, 6])
def test_add(a, b):
    assert mymath.add(a, b) > 0

Testing the isalnum method

The stdtypes have a number of methods that return a boolean. Such functions are excellent to demonstrate parametrization.

import pytest


@pytest.mark.parametrize("val", ["a", "b", "א", "2"])
def test_isalnum(val):
    assert val.isalnum()

@pytest.mark.parametrize("val", ["_", " "])
def test_not_isalnum(val):
    assert not val.isalnum()

cpython tests

Parametrize PyTest with multiple parameters

import pytest

@pytest.mark.parametrize("name,email", [
    ("Foo", "foo@email.com"),
    ("Bar", "bar@email.com"),
])
def test_cases(name, email):
    print(f"name={name}  email={email}")
    assert email.lower().startswith(name.lower())

Output:

========= test session starts =======
platform linux -- Python 3.7.3, pytest-5.3.2, py-1.8.0, pluggy-0.13.0
rootdir: /home/gabor/work/slides/python-programming/examples/pytest
plugins: flake8-1.0.4
collected 2 items

test_with_params.py name=Foo  email=foo@email.com
.name=Bar  email=bar@email.com
.

========== 2 passed in 0.01s =======

Exercise: parametrize the tests of the math functions

In this example we have several test cases in each test function. That means if one of the assertions fail the whole function fails and we don’t know what would be the result of the other test assertions.

import math

def test_gcd():
    assert math.gcd(6, 9) == 3
    assert math.gcd(17, 9) == 1

def test_ceil():
    assert math.ceil(0) == 0
    assert math.ceil(0.1) == 1
    assert math.ceil(-0.1) == 0

def test_factorial():
    assert math.factorial(0) == 1
    assert math.factorial(1) == 1
    assert math.factorial(2) == 2
    assert math.factorial(3) == 6

We could split up the test cases to several test functions along this example:

import math

def test_gcd_6_9():
    assert math.gcd(6, 9) == 3

def test_gcd_17_9():
    assert math.gcd(17, 9) == 1

def test_ceil_0():
    assert math.ceil(0) == 0

def test_ceil_0_1():
    assert math.ceil(0.1) == 1

def test_ceil__0_1():
    assert math.ceil(-0.1) == 0

def test_factorial_0():
    assert math.factorial(0) == 1

def test_factorial_1():
    assert math.factorial(1) == 1

def test_factorial_2():
    assert math.factorial(2) == 2

def test_factorial_3():
    assert math.factorial(3) == 6

Solution: Pytest test math functions

import math
import pytest

def test_gcd():
    assert math.gcd(6, 9) == 3
    assert math.gcd(17, 9) == 1

@pytest.mark.parametrize("val,expected", [
    (0, 0),
    (0.1, 1),
    (-0.1, 0),
    ])
def test_ceil(val, expected):
    assert math.ceil(val) == expected

@pytest.mark.parametrize("n,expected", [
    (0, 1),
    (1, 1),
    (2, 2),
    (3, 6),
    ])
def test_factorial(n, expected):
    assert math.factorial(n) == expected

Pytest assert

PyTest: failure reports

  • Reporting success is boring.
  • Reporting failure can be interesting: assert + introspection

One hopes that most of the time most of the tests pass and only very few fail. We are generally not interested why and how tests pass. They should do that silently.

We are interested how tests fail. What was the expected value, what is the actual value?

It is easy to see this if the expected result is a small number or a short string, but what if the expected result is a long string or a complex data structure with lots of values? What if we cannot expect exact results?

PyTest: compare numbers

def double(n):
    #return 2*n
    return 2+n

def test_double():
    assert double(2) == 4
    assert double(21) == 42

$ pytest -q test_number_equal.py

F                                                                        [100%]
=================================== FAILURES ===================================
_________________________________ test_double __________________________________

    def test_double():
        assert double(2) == 4
>       assert double(21) == 42
E       assert 23 == 42
E        +  where 23 = double(21)

test_number_equal.py:7: AssertionError
=========================== short test summary info ============================
FAILED test_number_equal.py::test_double - assert 23 == 42
1 failed in 0.09s

In this case it is easy to see what is the difference between the expected and the actual value.

Well except that pytest does not have a notion of which side is the expected side and which is the actual value, but you can clearly see if from the code.

PyTest: compare numbers relatively

def get_number():
    return 23

def test_get_number():
    assert get_number() < 0

$ pytest -q test_number_less_than.py

Output:

F                                                                        [100%]
=================================== FAILURES ===================================
_______________________________ test_get_number ________________________________

    def test_get_number():
>       assert get_number() < 0
E       assert 23 < 0
E        +  where 23 = get_number()

test_number_less_than.py:5: AssertionError
=========================== short test summary info ============================
FAILED test_number_less_than.py::test_get_number - assert 23 < 0
1 failed in 0.09s

PyTest: compare strings

Comparing short strings also work relatively well as we humans will be able to read the error report.

def get_string():
    return "abc"

def test_get_string():
    assert get_string() == "abd"


$ pytest -q test_string_equal.py

Output:

F                                                                        [100%]
=================================== FAILURES ===================================
_______________________________ test_get_string ________________________________

    def test_get_string():
>       assert get_string() == "abd"
E       AssertionError: assert 'abc' == 'abd'
E         
E         - abd
E         + abc

test_string_equal.py:5: AssertionError
=========================== short test summary info ============================
FAILED test_string_equal.py::test_get_string - AssertionError: assert 'abc' =...
1 failed in 0.04s

PyTest: compare long strings

However, comparing long strings would be extremely difficult for us.

Luckily pytest will point us to the first character that differs.

import string

def get_string(s):
    return string.printable + s + string.printable

def test_long_strings():
    assert get_string('a') == get_string('b')


$ pytest -q test_long_strings.py

Output:

    def test_long_strings():
>       assert get_string('a') == get_string('b')
E       AssertionError: assert '0123456789ab...t\n\r\x0b\x0c' == '0123456789abc...t\n\r\x0b\x0c'
E         Skipping 90 identical leading characters in diff, use -v to show
E         Skipping 91 identical trailing characters in diff, use -v to show
E           {|}~
E
E         - a012345678
E         ? ^
E         + b012345678
E         ? ^

PyTest: is one string in another strings?

Well, the fact the one string is not part of another string is rather difficult to show. Pytest will show ~250 characters.

import string

def get_string():
    return string.printable * 30

def test_long_strings():
    assert 'hello' in get_string()


pytest -q test_substring.py

Output


    def test_long_strings():
>       assert 'hello' in get_string()
E       assert 'hello' in '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c012345...x0b\x0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'
E        +  where '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c012345...x0b\x0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c' = get_string()

PyTest test any expression

def test_expression_equal():
    a = 3
    assert a % 2 == 0

$ pytest -q test_expression_equal.py

Output

F                                                                        [100%]
=================================== FAILURES ===================================
____________________________ test_expression_equal _____________________________

    def test_expression_equal():
        a = 3
>       assert a % 2 == 0
E       assert (3 % 2) == 0

test_expression_equal.py:3: AssertionError
=========================== short test summary info ============================
FAILED test_expression_equal.py::test_expression_equal - assert (3 % 2) == 0
1 failed in 0.09s

PyTest element in list

def get_list():
    return ["monkey", "cat"]

def test_in_list():
    assert "dog" in get_list()


$ pytest test_in_list.py

    def test_in_list():
>       assert "dog" in get_list()
E       AssertionError: assert 'dog' in ['monkey', 'cat']
E        +  where ['monkey', 'cat'] = get_list()

PyTest compare short lists

import string
import re

def get_lista():
    return 'a', 'b', 'c'
def get_listx():
    return 'x', 'b', 'y'

def test_short_lists():
    assert get_lista() == get_listx()

$ pytest test_short_lists.py

    def test_short_lists():
>       assert get_lista() == get_listx()
E       AssertionError: assert ('a', 'b', 'c') == ('x', 'b', 'y')
E         At index 0 diff: 'a' != 'x'
E         Use -v to get the full diff

PyTest compare short lists - verbose output

$ pytest -v test_short_lists.py

    def test_short_lists():
>       assert get_lista() == get_listx()
E       AssertionError: assert ('a', 'b', 'c') == ('x', 'b', 'y')
E         At index 0 diff: 'a' != 'x'
E         Full diff:
E         - ('a', 'b', 'c')
E         ?   ^         ^
E         + ('x', 'b', 'y')
E         ?   ^         ^

PyTest compare lists

import string
import re

def get_list(s):
    return list(string.printable + s + string.printable)

def test_long_lists():
    assert get_list('a') == get_list('b')

$ pytest test_lists.py

    def test_long_lists():
>       assert get_list('a') == get_list('b')
E       AssertionError: assert ['0', '1', '2...'4', '5', ...]
            == ['0', '1', '2'...'4', '5', ...]
E         At index 100 diff: 'a' != 'b'
E         Use -v to get the full diff

PyTest compare dictionaries - different values

def test_different_value():
    a = {
        "name" : "Whale",
        "location": "Ocean",
        "size": "huge",
    }
    b = {
        "name" : "Whale",
        "location": "Water",
        "size": "huge",
    }
    assert a == b

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest
plugins: flake8-1.0.6, dash-1.17.0
collected 1 item

test_dictionaries.py F                                                   [100%]

=================================== FAILURES ===================================
_____________________________ test_different_value _____________________________

    def test_different_value():
        a = {
            "name" : "Whale",
            "location": "Ocean",
            "size": "huge",
        }
        b = {
            "name" : "Whale",
            "location": "Water",
            "size": "huge",
        }
>       assert a == b
E       AssertionError: assert {'location': ...size': 'huge'} == {'location': ...size': 'huge'}
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'location': 'Ocean'} != {'location': 'Water'}
E         Use -v to get the full diff

test_dictionaries.py:12: AssertionError
=========================== short test summary info ============================
FAILED test_dictionaries.py::test_different_value - AssertionError: assert {'...
============================== 1 failed in 0.03s ===============================

PyTest compare dictionaries - missing-keys

def test_missing_key():
    a = {
        "name" : "Whale",
        "size": "huge",
    }
    b = {
        "name" : "Whale",
        "location": "Water",
    }
    assert a == b

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest
plugins: flake8-1.0.6, dash-1.17.0
collected 1 item

test_dictionaries_missing_keys.py F                                      [100%]

=================================== FAILURES ===================================
_______________________________ test_missing_key _______________________________

    def test_missing_key():
        a = {
            "name" : "Whale",
            "size": "huge",
        }
        b = {
            "name" : "Whale",
            "location": "Water",
        }
>       assert a == b
E       AssertionError: assert {'name': 'Wha...size': 'huge'} == {'location': ...ame': 'Whale'}
E         Omitting 1 identical items, use -vv to show
E         Left contains 1 more item:
E         {'size': 'huge'}
E         Right contains 1 more item:
E         {'location': 'Water'}
E         Use -v to get the full diff

test_dictionaries_missing_keys.py:10: AssertionError
=========================== short test summary info ============================
FAILED test_dictionaries_missing_keys.py::test_missing_key - AssertionError: ...
============================== 1 failed in 0.03s ===============================

PyTest Fixtures

PyTest: What are Fixtures?

  • In generally we call test fixture the environment in which a test is expected to run.
  • Pytest uses the same word for a more generic concept. All the techniques that make it easy to set up the environment and to tear it down after the tests.

General examples:

  • Setting up a database server - then removing it to clean the machine.
  • Maybe filling the database server with some data - emptying the database.

Specific examples:

  • If I’d like to test the login mechanism, I need that before the test starts running we’ll have a verified account in the system.
  • If I test the 3rd element in a pipeline I need the results of the 2nd pipeline to get started and after the test runs I need to remove all those files.

Fixtures in Pytest

There are two major ways to use fixtures in Python:

  • The more traditional xUnit-style.
  • The pytest-style.

We’ll see both of them.

PyTest: test with functions

If we don’t have any of the fixture services we need to write a lot of code.

We want to setup the database server before the first test. However the user can run all the test or a single one of the tests. So how can we know when to call the setup_db_server function? We don’t know. So we call it at the beginning of each test-function and then inside the the setup_db_server we store the name of the db_server. If we already have it then we don’t try to set it up again.

We would also want to tear down the db server after the last test, but we don’t know when to call it. If we call at the end of each test-function then it might run more than once. So we commented our teardown_db_server().

We would like to call the setup_db() before every test. That requires including it in every test-function, but that works.

We would like to call the teardown_db() after every test. That works for successful test-function, but if the test-function raises an exception (e.g. test_two has assert False) then the teardown_db() won’t be called.

So this soltion is very partial.

import tempfile
import time

def test_one():
    db_server = setup_db_server()
    db = setup_db()
    print(f"    test_one         {db}")
    assert True
    print("    test_one after")
    teardown_db(db)
    # teardown_db_server(db_server)

def test_two():
    db_server = setup_db_server()
    db = setup_db()
    print(f"    test_two         {db}")
    assert False
    print("    test_two after")
    teardown_db(db)
    # teardown_db_server(db_server)

def test_three():
    db_server = setup_db_server()
    db = setup_db()
    print(f"    test_three       {db}")
    assert True
    print("    test_three after")
    teardown_db(db)
    # teardown_db_server(db_server)

def setup_db():
    db = str(time.time()).replace(".", "_")
    print(f"setup_db             {db}")
    return db

def teardown_db(db):
    print(f"teardown_db          {db}")


def setup_db_server():
    print("setup db_server")
    if 'db_server' not in setup_db_server.__dict__:
        setup_db_server.db_server = tempfile.TemporaryDirectory()
        setup_db_server.time = time.time()
        print(f"new   db_server environment {setup_db_server.db_server.name}")
    return setup_db_server.db_server

def teardown_db_server(db_server):
    print("teardown_db_server {setup_db_server.db_server.name}")

PyTest: test with functions outpus

setup db_server
new   db_server environment /tmp/tmp2hxf25v3
setup_db             1773757447_0637202
    test_one         1773757447_0637202
    test_one after
teardown_db          1773757447_0637202
.setup db_server
setup_db             1773757447_064304
    test_two         1773757447_064304
Fsetup db_server
setup_db             1773757447_0757604
    test_three       1773757447_0757604
    test_three after
teardown_db          1773757447_0757604
.
=================================== FAILURES ===================================
___________________________________ test_two ___________________________________

    def test_two():
        db_server = setup_db_server()
        db = setup_db()
        print(f"    test_two         {db}")
>       assert False
E       assert False

test_functions.py:17: AssertionError
=========================== short test summary info ============================
FAILED test_functions.py::test_two - assert False
1 failed, 2 passed in 0.06s

PyTest Fixture setup and teardown xUnit style

  • setup_function
  • teardown_function
  • setup_module
  • teardown_module

There are two mechanism in PyTest to setup and teardown fixtures. One of them is the xUnit-style system that is also available in other languages such as Java and C#.

In this example there are 3 tests, 3 functions that are called test_SOMETHING. There are also two pairs of functions to setup and teardown the fixtures on a per-function and per-module level.

Before starting to run the tests of this file PyTest will run the setup_module function, and after it is done running the tests PyTest will run the teardonw_module function. This will happen even if one or more of the tests failed. These functions will be called once regardless of the number of tests we have in the module.

Before every test function PyTest will run the setup_function and after the test finished it will run the teardown_function. Regardless of the success or failure of the test.

So in our case where we have all 4 of the fixture functions implemented and we have 3 tests function the order can be seen on the next page.

In this example we also see one of the major issues of this style. The variable db that is set in the setup_module function must be marked as global in order to make it accessible in the test functions and in the teardown_module function.

import tempfile
import time

def setup_module():
    global db_server
    db_server = tempfile.TemporaryDirectory()
    print(f"setup_module:         {db_server.name}")

def teardown_module():
    print(f"teardown_module       {db_server.name}")


def setup_function():
    global db
    db = time.time()
    print(f"  setup_function                                              {db}")

def teardown_function():
    print(f"  teardown_function                                          {db}")


def test_one():
    print(f"    test_one          {db_server.name} {db}")
    assert True
    print("    test_one after")

def test_two():
    print(f"    test_two          {db_server.name} {db}")
    assert False
    print("    test_two after")

def test_three():
    print(f"    test_three        {db_server.name} {db}")
    assert True
    print("    test_three after")

PyTest Fixture setup and teardown output

test_fixture.py .F.

$ pytest -sq test_fixture.py

Output:

setup_module:         /tmp/tmp5vfj_uuh
  setup_function                                              1773757663.796925
    test_one          /tmp/tmp5vfj_uuh 1773757663.796925
    test_one after
.  teardown_function                                          1773757663.796925
  setup_function                                              1773757663.797643
    test_two          /tmp/tmp5vfj_uuh 1773757663.797643
F  teardown_function                                          1773757663.797643
  setup_function                                              1773757663.8085847
    test_three        /tmp/tmp5vfj_uuh 1773757663.8085847
    test_three after
.  teardown_function                                          1773757663.8085847
teardown_module       /tmp/tmp5vfj_uuh

=================================== FAILURES ===================================
___________________________________ test_two ___________________________________

    def test_two():
        print(f"    test_two          {db_server.name} {db}")
>       assert False
E       assert False

test_fixture.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_fixture.py::test_two - assert False
1 failed, 2 passed in 0.06s

Note, the teardown_function is executed even after failed tests.

PyTest: Fixture Class setup and teardown

  • setup_class
  • teardown_class
  • setup_method
  • teardown_method

In case you are using test classes then you can use another 2 pairs of functions, well actually methods, to setup and teardown the environment. In this case it is much easier to pass values from the setup to the test functions and to the teardown function, but we need to write the whole thing in OOP style.

Also note, the test functions are independent. They all see the attributes set in the setup_class, but the test functions cannot pass values to each other.

class TestClass():
    def setup_class(self):
        print("setup_class called once for the class")
        print(self)
        self.db = "mydb"
        self.test_counter = 0

    def teardown_class(self):
        print(f"teardown_class called once for the class {self.db}")

    def setup_method(self):
        self.test_counter += 1
        print(f"  setup_method called for every method {self.db} {self.test_counter}")
        print(self)

    def teardown_method(self):
        print(f"  teardown_method called for every method {self.test_counter}")


    def test_one(self):
        print("    one")
        assert True
        print("    one after")

    def test_two(self):
        print("    two")
        assert False
        print("    two after")

    def test_three(self):
        print("    three")
        assert True
        print("    three after")

PyTest: Fixture Class setup and teardown output

$ pytest -sq test_class.py

Output:

setup_class called once for the class
<class 'test_class.TestClass'>
  setup_method called for every method 1
<test_class.TestClass object at 0x7d5621762960>
    one
    one after
.  teardown_method called for every method 1
  setup_method called for every method mydb 1
<test_class.TestClass object at 0x7d5620ee6060>
    two
F  teardown_method called for every method 1
  setup_method called for every method 1
<test_class.TestClass object at 0x7d5620c1c830>
    three
    three after
.  teardown_method called for every method 1
teardown_class called once for the class mydb

What is Dependency injection?

def serve_bolognese(pasta, sauce):
    dish = mix(pasta, sauce)
    return dish
  1. Find function.
  2. Check parameters of the function.
  3. Prepare the appropriate objects.
  4. Call the function passing these objects.

Pytest fixture - tmpdir

  • Probably the simples fixture that PyTest provides is the tmpdir.
  • Pytest will prepare a temporary directory and call the test function passing the path to the tmpdir.
  • PyTest will also clean up the temporary folder, though it will keep the 3 most recent ones. (this is configurable)
  • tmp_path is pathlib.Path.
import os


def test_something(tmpdir):
    print(tmpdir)      # /private/var/folders/ry/z60xxmw0000gn/T/pytest-of-gabor/pytest-14/test_read0
    print(type(tmpdir)) # _pytest._py.path.LocalPath

    d = tmpdir.mkdir("subdir")
    fh = d.join("config.ini")
    fh.write("Some text")

    filename = os.path.join( fh.dirname, fh.basename )

    # ...

pytest -sq test_tmpdir.py

Pytest and tmpdir

  • This is a simple application that reads and writes config files (ini file).
  • We can test the parse_file by preparing some input files and check if we get the expected data structure.
  • In order to test the save_file we need to be able to save a file somewhere.
  • Saving it in the current folder will create garbage files. (and the folder might be read-only in some environments).
  • For each test we’ll have to come up with a separate filename so they won’t collide.
  • Using a tmpdir solves this problem.
import re

def parse_file(filename):
    data = {}
    with open(filename) as fh:
        for row in fh:
            row = row.rstrip("\n")
            if re.search(r'=', row):
                k, v = re.split(r'\s*=\s*', row)
                data[k] = v
            else:
                pass # error reporting?
    return data

def save_file(filename, data):
    with open(filename, 'w') as fh:
        for k in data:
            fh.write("{}={}\n".format(k, data[k]))

if __name__ == '__main__':
    print(parse_file('a.cfg'))

name=Foo Bar
email  =   foo@bar.com


import mycfg
import os

def test_parse():
    data = mycfg.parse_file('a.cfg')
    assert data, {
        'name'  : 'Foo Bar',
        'email' : 'foo@bar.com',
    }

def test_example(tmpdir):
    original = {
        'name'  : 'My Name',
        'email' : 'me@home.com',
        'home'  : '127.0.0.1',
    }
    filename = str(tmpdir.join('abc.cfg'))
    assert not os.path.exists(filename)
    mycfg.save_file(filename, original)
    assert os.path.exists(filename)
    new = mycfg.parse_file(filename)
    assert new == original

Pytest CLI key-value store

  • This is a similar application - a file-base key-value store - where the data files is computed from the name of the program: store.json.
  • Running two tests in parallel will make the tests collide by using the same data file.
import os
import json

def set(key, value):
    data = _read_data()
    data[key] = value
    _save_data(data)

def get(key):
    data = _read_data()
    return(data.get(key))

def _save_data(data):
    filename = _get_filename()
    with open(filename, 'w') as fh:
        json.dump(data, fh, sort_keys=True, indent=4)


def _read_data():
    filename = _get_filename()
    data = {}
    if os.path.exists(filename):
        with open(filename) as fh:
            data = json.load(fh)
    return data

def _get_filename():
    path = os.path.dirname(os.path.abspath(__file__))
    filename = os.path.join(path, 'store.json')
    return filename


if __name__ == '__main__':
    import sys
    if len(sys.argv) == 3:
        cmd, key = sys.argv[1:]
        if cmd == 'get':
            print(get(key))
            exit(0)

    if len(sys.argv) == 4:
        cmd, key, value = sys.argv[1:]
        if cmd == 'set':
            set(key, value)
            print('SET')
            exit(0)
    print(f"""Usage:
           {sys.argv[0]} set key value
           {sys.argv[0]} get key
    """)

import store

def test_store():
    store.set('color', 'Blue')
    assert store.get('color') == 'Blue'

    store.set('color', 'Red')
    assert store.get('color') == 'Red'

    store.set('size', '42')
    assert store.get('size') == '42'
    assert store.get('color') == 'Red'

Pytest testing key-value store - environment variable

  • We need to be able to set the name of the data file externally. e.g. Using an environment variable.
  • This requires some change to the application to make it more testable.
  • Luckily this application already had a function called _get_filename.
import os
import json

def set(key, value):
    data = _read_data()
    data[key] = value
    _save_data(data)

def get(key):
    data = _read_data()
    return(data.get(key))

def _save_data(data):
    filename = _get_filename()
    with open(filename, 'w') as fh:
        json.dump(data, fh, sort_keys=True, indent=4)


def _read_data():
    filename = _get_filename()
    data = {}
    if os.path.exists(filename):
        with open(filename) as fh:
            data = json.load(fh)
    return data

def _get_filename():
    path = os.environ.get('STORE_DIR', os.path.dirname(os.path.abspath(__file__)))
    filename = os.path.join(path, 'store.json')
    return filename


if __name__ == '__main__':
    import sys
    if len(sys.argv) == 3:
        cmd, key = sys.argv[1:]
        if cmd == 'get':
            print(get(key))
            exit(0)

    if len(sys.argv) == 4:
        cmd, key, value = sys.argv[1:]
        if cmd == 'set':
            set(key, value)
            print('SET')
            exit(0)
    print(f"""Usage:
           {sys.argv[0]} set key value
           {sys.argv[0]} get key
    """)

import store
import os

def test_store(tmpdir):
    os.environ['STORE_DIR'] = str(tmpdir) # str expected, not LocalPath
    print(tmpdir)
    store.set('color', 'Blue')
    assert store.get('color') == 'Blue'

    store.set('color', 'Red')
    assert store.get('color') == 'Red'

    store.set('size', '42')
    assert store.get('size') == '42'
    assert store.get('color') == 'Red'

Pytest testing key-value store - environment variable (outside)

import os
import json

path = os.environ.get('STORE_DIR', os.path.dirname(os.path.abspath(__file__)))
filename = os.path.join(path, 'store.json')

def set(key, value):
    data = _read_data()
    data[key] = value
    _save_data(data)

def get(key):
    data = _read_data()
    return(data.get(key))

def _save_data(data):
    with open(filename, 'w') as fh:
        json.dump(data, fh, sort_keys=True, indent=4)


def _read_data():
    data = {}
    if os.path.exists(filename):
        with open(filename) as fh:
            data = json.load(fh)
    return data



if __name__ == '__main__':
    import sys
    if len(sys.argv) == 3:
        cmd, key = sys.argv[1:]
        if cmd == 'get':
            print(get(key))
            exit(0)

    if len(sys.argv) == 4:
        cmd, key, value = sys.argv[1:]
        if cmd == 'set':
            set(key, value)
            print('SET')
            exit(0)
    print(f"""Usage:
           {sys.argv[0]} set key value
           {sys.argv[0]} get key
    """)

import os
import store

def test_store(tmpdir):
    os.environ['STORE_DIR'] = str(tmpdir) # str expected, not LocalPath
    print(os.environ['STORE_DIR'])
    store.set('color', 'Blue')
    assert store.get('color') == 'Blue'

    store.set('color', 'Red')
    assert store.get('color') == 'Red'

    store.set('size', '42')
    assert store.get('size') == '42'
    assert store.get('color') == 'Red'

Application that prints to STDOUT and STDERR

A very simple function that print to the screen. How can we test this?

An even wors situation is when a function both does some (computational) work and prints to the screen, something not ideal from a design point of view. However even if the sole job of a function is to print to the screen, we still need a way to test it.

import sys

def welcome(to_out, to_err=None):
    print(f"STDOUT: {to_out}")
    if to_err:
        print(f"STDERR: {to_err}", file=sys.stderr)

We can write a separate program that uses this function and “eyeball” the result, but we can do better.

from greet import welcome

welcome("Jane", "Joe")
print('---')
welcome("Becky")

Output:

STDERR: Joe
STDOUT: Jane
---
STDOUT: Becky

Pytest capture STDOUT and STDERR with capsys

The capsys fixture captures everything that is printed to STDOUT and STDERR so we can compare that to the expected output and error.

  • We need to include capsys as the parameter of the test function.
  • The first call to readouterr will return whatever was printed to STDOUT and STDERR respectively from the start of the test function.
  • The subsequent calls to readouterr will return the output that was generated since the previous call to readouterr.
from greet import welcome

def test_myoutput(capsys):
    welcome("hello", "world")
    out, err = capsys.readouterr()
    assert out == "STDOUT: hello\n"
    assert err == "STDERR: world\n"

    welcome("next")
    out, err = capsys.readouterr()
    assert out == "STDOUT: next\n"
    assert err == ""

pytest test_greet.py

Note, however, using -s won’t print anything extra in this case as the ourput was captured by capsys.

pytest -s test_greet.py

PyTest - write your own fixture

  • tmpdir and capsys are nice to have, but we will need more complex setup and teardown.

  • We can write any function to become fixture, we only need to decorate it with @pytest.fixture

  • We can implement fixture functions to act like the xUnit fixture we saw earlier or using dependency injection as tmpdir and capsys work.

Pytest Fixture - autouse fixtures (using yield)

Using the @pytest.fixture decorator we can designate some function to be called automatically before and/or after each test.

If we set the scope to module then the fixtures will be similar to the setup_module, teardown_module functions of the xUnit style.

If we set thescope to function then the fixtures will be similar to the setup_function, teardown_function functions of the xUnit style.

import pytest
import time

@pytest.fixture(autouse = True, scope="module")
def fix_module():
    answer = 42
    print(f"Module setup {answer}")
    yield
    print(f"Module teardown {answer}")


@pytest.fixture(autouse = True, scope="function")
def fix_function():
    start = time.time()
    print(f"  Function setup {start}")
    yield
    print(f"  Function teardown {start}")


def test_one():
    print("    Test one")
    assert True
    print("    Test one - 2nd part")

def test_two():
    print("    Test two")
    assert False
    print("    Test two - 2nd part")

Output:

Module setup 42
  Function setup 1612427609.9726396
    Test one
    Test one - 2nd part
  Function teardown 1612427609.9726396
  Function setup 1612427609.9741583
    Test two
  Function teardown 1612427609.9741583
Module teardown 42

Share fixtures among test files: conftest.py

You can create a file called conftest.py and place it in the root of the test folder or the root of the project if you don’t have a test-folder. Pytest will automaticall import its functions into every test.

import pytest

@pytest.fixture(autouse = True, scope="session")
def fix_session():
    print("\nSession setup")
    yield
    print("\nSession teardown")


@pytest.fixture(autouse = True, scope="module")
def fix_module():
    print("\n  Module setup")
    yield
    print("\n  Module teardown")


@pytest.fixture(autouse = True, scope="function")
def fix_function():
    print("\n    Function setup")
    yield
    print("\n    Function teardown")

def test_one():
    print("      Test Blue one")
    assert True


def test_two():
    print("      Test Blue two")
    assert False

def test_three():
    print("      Test Green Three")
    assert True

pytest -qs

Output:


Session setup

  Module setup
    Function setup
      Test Blue one
    Function teardown

    Function setup
      Test Blue two
    Function teardown
  Module teardown

  Module setup
    Function setup
      Test Green Three
    Function teardown
  Module teardown

Session teardown

Manual fixtures (using dependency injection)

If we don’t set the autouse to True then by default it is False. Meaning you will have to explicitly use them. However instead of calling these function we use their names as parameters of our test functions. Pytest will notice that your test function requires certain parameters, it will find the fixture with the given name, call it, and set the variable to the value “returned” by the fixture.

import pytest

@pytest.fixture()
def blue():
   print("Blue setup")
   yield
   print("Blue teardown")

@pytest.fixture()
def green():
   print("Green setup")
   yield
   print("Green teardown")

#def test_try(yellow):
#    print("yellow")

def test_one(blue, green):
   print("    Test one")


def test_two(green, blue):
   print("    Test two")
   assert False

Output:

Blue setup
Green setup
    Test one
Green teardown
Blue teardown

Green setup
Blue setup
    Test two
Blue teardown
Green teardown
  • We can’t add fixtures to test_functions as decorators (as I was the case in NoseTest), we need to use dependency injection.

Pytest Fixture providing value

import pytest
import application


@pytest.fixture()
def app():
    print('app starts')
    myapp = application.App()
    return myapp


def test_add_user_foo(app):
    app.add_user("Foo")
    assert app.get_user() == 'Foo'

def test_add_user_bar(app):
    app.add_user("Bar")
    assert app.get_user() == 'Bar'


class App:
    def __init__(self):
        self.pi = 3.14
        # .. set up database
        print("__init__ of App")


    def add_user(self, name):
        print("Working on add_user({})".format(name))
        self.name = name

    def get_user(self):
        return self.name

$ pytest -sq

Output:

getapp starts
__init__ of App
Working on add_user(Foo)

getapp starts
__init__ of App
Working on add_user(Bar)

Pytest Fixture providing value with teardown

import pytest
import application


@pytest.fixture()
def myapp():
    print('myapp starts')
    app = application.App()

    yield app

    app.shutdown()
    print('myapp ends')

def test_add_user_foo(myapp):
    myapp.add_user("Foo")
    assert myapp.get_user() == 'Foo'

def test_add_user_bar(myapp):
    myapp.add_user("Bar")
    assert myapp.get_user() == 'Bar'


class App:
    def __init__(self):
        self.pi = 3.14
        # .. set up database
        print("__init__ of App")


    def shutdown(self):
        print("shutdown of App cleaning up database")


    def add_user(self, name):
        print("Working on add_user({})".format(name))
        #self.name = name

    def get_user(self):
        return self.name

$ pytest -sq

Output:

getapp starts
__init__ of App
Working on add_user(Foo)
shutdown of App cleaning up database
getapp ends


getapp starts
__init__ of App
Working on add_user(Bar)
shutdown of App cleaning up database
getapp ends

Pytest create fixture with file(s) - app and test

import json
import os


def _get_config_file():
    return os.environ.get('APP_CONFIG_FILE', 'config.json')

def _read_config():
    config_file = _get_config_file()
    with open(config_file) as fh:
        return json.load(fh)

def app(protocol):
    config = _read_config()
    # ... do stuff based on the config
    address = protocol + '://' + config['host'] + ':' + config['port']

    path = os.path.dirname(_get_config_file())
    outfile = os.path.join(path, 'out.txt')
    with open(outfile, 'w') as fh:
        fh.write(address)

    return address


{"host" : "szabgab.com", "port" : "80"}

from myapp import app
result = app('https')
print(result)

https://szabgab.com:80
  • Test application
import json
import os

from myapp import app

def test_app_one(tmpdir):
    config_file = os.path.join(str(tmpdir), 'conf.json')
    with open(config_file, 'w') as fh:
        json.dump({'host' : 'code-maven.com', 'port' : '443'}, fh)
    os.environ['APP_CONFIG_FILE'] = config_file

    result = app('https')

    assert result == 'https://code-maven.com:443'
    outfile = os.path.join(str(tmpdir), 'out.txt')
    with open(outfile) as fh:
        output_in_file = fh.read()
    assert output_in_file == 'https://code-maven.com:443'


def test_app_two(tmpdir):
    config_file = os.path.join(str(tmpdir), 'conf.json')
    with open(config_file, 'w') as fh:
        json.dump({'host' : 'code-maven.com', 'port' : '443'}, fh)
    os.environ['APP_CONFIG_FILE'] = config_file

    result = app('http')

    assert result == 'http://code-maven.com:443'
    outfile = os.path.join(str(tmpdir), 'out.txt')
    with open(outfile) as fh:
        output_in_file = fh.read()
    assert output_in_file == 'http://code-maven.com:443'

Pytest create fixture with file(s) - helper function

import json
import os

from myapp import app

def test_app_one(tmpdir):
    setup_config(tmpdir)

    result = app('https')

    assert result == 'https://code-maven.com:443'
    output_in_file = read_file(tmpdir)
    assert output_in_file == 'https://code-maven.com:443'

def test_app_two(tmpdir):
    setup_config(tmpdir)

    result = app('http')

    assert result == 'http://code-maven.com:443'
    output_in_file = read_file(tmpdir)
    assert output_in_file == 'http://code-maven.com:443'

def setup_config(tmpdir):
    config_file = os.path.join(str(tmpdir), 'conf.json')
    with open(config_file, 'w') as fh:
        json.dump({'host' : 'code-maven.com', 'port' : '443'}, fh)
    os.environ['APP_CONFIG_FILE'] = config_file


def read_file(tmpdir):
    outfile = os.path.join(str(tmpdir), 'out.txt')
    with open(outfile) as fh:
        output_in_file = fh.read()
    return output_in_file

Pytest create fixture with file(s) - fixture

import pytest
import json
import os

from myapp import app

def test_app_one(outfile):
    result = app('https')

    assert result == 'https://code-maven.com:443'
    output_in_file = read_file(outfile)
    assert output_in_file == 'https://code-maven.com:443'

def test_app_two(outfile):
    result = app('http')

    assert result == 'http://code-maven.com:443'
    output_in_file = read_file(outfile)
    assert output_in_file == 'http://code-maven.com:443'


@pytest.fixture()
def outfile(tmpdir):
    #print(tmpdir)
    config_file = os.path.join(str(tmpdir), 'conf.json')
    with open(config_file, 'w') as fh:
        json.dump({'host' : 'code-maven.com', 'port' : '443'}, fh)
    os.environ['APP_CONFIG_FILE'] = config_file
    return os.path.join(str(tmpdir), 'out.txt')


def read_file(outfile):
    with open(outfile) as fh:
        output_in_file = fh.read()
    return output_in_file

Pytest fixture inject data

import pytest

@pytest.fixture()
def config():
    return {
       'name'  : 'Foo Bar',
       'email' : 'foo@bar.com',
    }

def test_some_data(config):
    assert True
    print(config)

Pytest fixture for MongoDB

# conftest.py

import pytest
import os, time

from app.common import get_mongo

@pytest.fixture(autouse = True)
def configuration():
    dbname = 'test_app_' + str(int(time.time()))
    os.environ['APP_DB_NAME'] = dbname

    yield

    get_mongo().drop_database(dbname)

Pytest parametrized fixture

Sometimes we would like to pass some parameters to the fixture. We can do this with one or more parameters.

import os
import pathlib
import time
import pytest


@pytest.fixture(autouse = True, scope="function", params=["name"])
def generate(name):
    print(f"Fixture before test using {name}")
    yield
    print(f"Fixture after test using {name}")

@pytest.mark.parametrize("name", ["apple"])
def test_with_param(name):
    print(f"Test using {name}")

@pytest.mark.parametrize("name", ["banana"])
def test_without_param():
    print(f"Test not using param")

Output:

Fixture before test using apple
Test using apple
Fixture after test using apple

Fixture before test using banana
Test not using param
Fixture after test using banana

Pytest parametrized fixture with dependency injection

import os
import pathlib
import time
import pytest


@pytest.fixture(params=["name"])
def generate(name):
    print(f"Fixture before test using {name}")
    yield
    print(f"Fixture after test using {name}")

@pytest.mark.parametrize("name", ["apple"])
def test_with_param(name, generate):
    print(f"Test using {name}")

@pytest.mark.parametrize("name", ["banana"])
def test_without_param(generate):
    print(f"Test not using param")

@pytest.mark.parametrize("name", ["banana"])
def test_with_param_without_fixture(name):
    print(f"Test using param {name} and not using fixture")

def test_without_param_without_fixture():
    print(f"Test not using param and not using fixture")

Output:

Fixture before test using apple
Test using apple
Fixture after test using apple
Fixture before test using banana
Test not using param
Fixture after test using banana

Pytest parametrized fixture to use Docker

I created a GitHub Action for the OSDC site generator which is running inside a Docker container. At one point I wanted the whole image creation and running in the image be part of the test.

import os
import pathlib
import time
import pytest


@pytest.fixture(autouse = True, scope="function", params=["name"])
def generate(name):
    image = f"osdc-test-{str(time.time())}"
    os.system(f'docker build -t {image} .')
    os.system(f'docker run --rm -w /data -v{os.getcwd()}/{name}:/data  {image}')
    yield
    os.system(f'docker rmi {image}')

@pytest.mark.parametrize("name", ["test1"])
def test_one(name):
    root = pathlib.Path(name)
    site = root.joinpath('_site')
    assert site.exists()
    assert site.joinpath('index.html').exists()
    pages = site.joinpath('osdc-skeleton')
    assert pages.exists()

    with pages.joinpath('about.html').open() as fh:
        html = fh.read()
    assert '<title>About</title>' in html

Pytest Mocking

Pytest: Mocking - why?

  • Testing environment that does not depend on external systems.
  • Faster tests (mock remote calls, mock whole databases, etc.)
  • Fake some code/application/API that does not exist yet.
  • Test error conditions in a system not under our control.
  • TDD, unit tests
  • Spaghetti code
  • Simulate hard to replicate cases
  • 3rd party APIs or applications

Pytest: Mocking - what?

  • Hard-coded path in code.
  • STDIN/STDOUT/STDERR.
  • External dependency (e.g. an API).
  • Random values.
  • Methods accessing a database.
  • Time.

Pytest: What is Mocking? - Test Doubles

Pytest: Monkey Patching

Pytest: Hard-coded path

In many organizations and in many projects at one point someone has decided that it is a good idea to have a fixed central folder or central configuration file.

import json

data_file = "/corporate/fixed/path/data.json"

def do_something():
    print(data_file)
    #with open(data_file) as fh:
    #    data = json.load(fh)
    #    ...

Pytest: Hard-coded path - testing

The problem with this is that when trying to test the application, the test will also use the same hard-coded path. Thus we cannot test with values that are different from what the corporation has. This might include the credentials of the production database.

Not good.

import app

def test_app_1():
    res = app.do_something()
    ...

def test_app_2():
    res = app.do_something()
    ...

pytest -s test_app.py

/corporate/fixed/path/data.json
./corporate/fixed/path/data.json
.
2 passed in 0.07s

Pytest: Hard-coded path - manually replace attribute

Because we are in Python, a dynamic language, we can access and replace the content of the data_file variable in the app module. This is sometimes called Monkey Patching. It works. Partially.

import app

def test_app_1():
    app.data_file = 'test_1.json'    # manually overwrite

    res = app.do_something()       # it is now test_1.json
    ...

def test_app_2():
    res = app.do_something()      # it is still test_1.json
    ...

We set the path in one of the test-functions, but it will be set in both.

pytest -v -s test_app_manually.py

test_1.json
.test_1.json
.
2 passed in 0.08s

Run test 1, we see the path we set:

pytest -v -s test_app_manually.py::test_app_1

test_1.json
.
1 passed in 0.07s

Run test 2, we see a the original path:

pytest -v -s test_app_manually.py::test_app_2

/corporate/fixed/path/data.json
.
1 passed in 0.07s

Pytest: Hard-coded path - monkeypatch attribute

We can also use the setattr method of the monkeypatch fixture.

The path is still rather fixed, but now it is fixed in the test. So we could prepare a few config file and use them in the tests.

import app

def test_app_1(monkeypatch):
    monkeypatch.setattr(app, 'data_file', 'test_1.json')

    res = app.do_something()    # It is now test_1.json
    ...

def test_app_2():
    res = app.do_something() # back to the original value
    ...

When running all the tests the first one will have the mocked filename the second one will have the hard-coded one.

$ pytest -s -q test_app_monkeypatch.py

test_1.json
./corporate/fixed/path/data.json
.
2 passed in 0.04s

When running the tests separately each one of them will have the same filename as they had when we ran them together.

$ pytest -s -q test_app_monkeypatch.py::test_app_1

test_1.json
.
1 passed in 0.04s

$ pytest -s -q test_app_monkeypatch.py::test_app_2

/corporate/fixed/path/data.json
.
1 passed in 0.04s

Pytest: Hard-coded path - monkeypatch attribute - tmpdir

An even better approach might be to use the tmpdir fixture to create a temporary folder, create the necessary file(s) there and use the monkeypatch fixture to point there.

import app

def test_app_1(monkeypatch, tmpdir):
    mocked_data_file = tmpdir.join('test_1.json')
    monkeypatch.setattr(app, 'data_file', mocked_data_file)

    res = app.do_something()
    ...

def test_app_2():
    res = app.do_something()    # back to the original value
    ...

/tmp/pytest-of-gabor/pytest-2/test_sum0/test_1.json
./corporate/fixed/path/data.json
.
2 passed in 0.04s

Pytest: Mocking slow external API call

We have a module called mymath (again) that does some computation for us. I think it uses the Pythagoras theorem to calculate the distance from the “origin”.

import externalapi

def compute(x, y):
    xx = externalapi.remote_compute(x)
    yy = externalapi.remote_compute(y)
    result = (xx+yy) ** 0.5
    return result

It uses an external API implemented in the externalapi.py file to compue the square of each number. Unfortunatelly this external API is slow.

import time

def remote_compute(x):
    time.sleep(5) # to imitate long running process
    return x*x

We can try to run the following code. It will take 10 seconds to compute this. When testing the mymath library we don’t want to wait 10 seconds for every test-case.

time python use_mymath.py

import mymath

print(mymath.compute(3, 4))

Pytest: Mocking slow external API call - manually replace function

We can create a function (we call it mocked_remote_compute) that we’ll use to replace the remote API call. However, ee don’t need to re-implement the whole complex algorithm of computing the square of numbers provided by the remote service. It is enough that we hard-code the responses we should get from the API for the values we are going to use in the tests.

Then we can manuall monkey-patch the method in the externalapi library.

import mymath

def mocked_remote_compute(x):
    print(f"mocked received {x}")
    if x == 3:
        return 9
    if x == 4:
        return 16
    raise Exception (f"The value {x} isn't supported by the mock")

mymath.externalapi.remote_compute = mocked_remote_compute

def test_compute():
    assert mymath.compute(3, 4) == 5

#def test_other():
#    res = mymath.compute(2, 7)
#    assert res == 7.28010988928

time pytest test_mymath.py

Pytest: Mocking slow external API call - monkeypatch

import mymath

def mocked_remote_compute(x):
    print(f"mocked received {x}")
    if x == 3:
        return 9
    if x == 4:
        return 16


def test_compute(monkeypatch):
    monkeypatch.setattr(mymath.externalapi, 'remote_compute', mocked_remote_compute)
    assert mymath.compute(3, 4) == 5
    ...

def test_other(monkeypatch):
    def mocked_remote_compute(x):
        print(f"other mocked received {x}")
        if x == 6:
            return 36
        if x == 8:
            return 64
    monkeypatch.setattr(mymath.externalapi, 'remote_compute', mocked_remote_compute)
    assert mymath.compute(6, 8) == 10
    ...

Pytest: Mocking broken external API response

How can we test the behaviour of our code on wrong response from the API?

In a way it is similar to testing our code with invalid or broken input, except the API call is initiated by our code and we don’t have control over the API service.

Unless, of course, we mock it.

So in this example we made the mock returned a string instead of a number. Our code raised a python-level exception. This is obviously not good. Now you need to decide whether you’d like to improve the application to check the values returned by the API or expect this test to fail?

import mymath
import pytest

def mocked_remote_compute(x):
    if x == 3:
        return '9'
    if x == 4:
        return 16

mymath.externalapi.remote_compute = mocked_remote_compute

# What should we really expect here?
# I don't want to see a Python-level exception
# Maybe an exception of our own.
# @pytest.mark.xfail(reason = "We currenty don't change the validity of the values returned by the API call.")
def test_compute_breaks():
    assert mymath.compute(3, 4) == 5

Pytest: Mocking STDIN

def ask_one():
    name = input("Please enter your name: ")
    print(f"Your name is {name}")

def ask_two():
    width = float(input("Please enter width: "))
    length = float(input("Please enter length: "))
    result = width * length
    print(f"{width}*{length} is {result}")

import app

app.ask_one()
#app.ask_two()

Pytest: Mocking STDIN manually mocking

import app
import io
import sys

def test_app(capsys):
    sys.stdin = io.StringIO('Foo')
    app.ask_one()
    out, err = capsys.readouterr()
    assert err == ''
    #print(out)
    assert out == 'Please enter your name: Your name is Foo\n'

def test_app_again(capsys):
    ...   # still the same handle

import app
import io
import sys

def test_app(capsys):
    sys.stdin = io.StringIO('3\n4')
    app.ask_two()
    out, err = capsys.readouterr()
    assert err == ''
    #print(out)
    assert out == 'Please enter width: Please enter length: 3.0*4.0 is 12.0\n'

Pytest: Mocking STDIN - monkeypatch

import app
import io
import sys

def test_one(capsys, monkeypatch):
    monkeypatch.setattr(sys, 'stdin', io.StringIO('Foo'))
    app.ask_one()
    out, err = capsys.readouterr()
    assert err == ''
    #print(out)
    assert out == 'Please enter your name: Your name is Foo\n'

def test_two(monkeypatch, capsys):
    monkeypatch.setattr(sys, 'stdin', io.StringIO('3\n4'))
    app.ask_two()
    out, err = capsys.readouterr()
    assert err == ''
    #print(out)
    assert out == 'Please enter width: Please enter length: 3.0*4.0 is 12.0\n'

Pytest: Mocking random numbers - the application

import random

class Game():
    def __init__(self):
        self.hidden = random.randint(1, 200)
        #print(hidden)

    def guess(self, guessed_number):
        if self.hidden == guessed_number:
            return 'match'
        if guessed_number < self.hidden:
            return 'too small'
        return 'too big'

Pytest: Mocking random numbers

from app import Game

game = Game()
while True:
    user_guess = input("Guess a number: ")
    if user_guess == "x":
        break
    if user_guess == "s":
        print(game.hidden)
        continue
    user_guess = int(user_guess)
    response = game.guess(user_guess)
    if response == 'match':
        print("matched")
        break
    print(response)

import app

def test_game(monkeypatch):
    monkeypatch.setattr(app.random, 'randint', lambda x, y: 70)
    game = app.Game()
    print(game.hidden)
    response = game.guess(100)
    assert response == 'too big'

    response = game.guess(50)
    assert response == 'too small'

    response = game.guess(70)
    assert response == 'match'

Pytest: Mocking multiple random numbers

import random

def random_sum(n):
    total = 0
    for _ in range(n):
        current = random.randint(0, 10)
        #print(current)
        total += current
    return total


import app

result = app.random_sum(3)
print(result)

import app

def test_random_sum(monkeypatch):
    values = [2, 3, 4]
    monkeypatch.setattr(app.random, 'randint', lambda _x, _y: values.pop(0))
    result = app.random_sum(3)
    assert result == 9

Pytest: Mocking time

There are several different problems with time

  • A login that should expire after 24 hours. We don’t want to wait 24 hours.
  • Some code that must be executed on certain dates. (e.g. January 1st every year)

Pytest: Mocking time (test expiration)

In this application the user can “login” by providing their name and then call the access_page method within session_length seconds.

Because we know it internally users the time.time function to retrieve the current time (in seconds since the epoch) we can replace that function with ours that will fake the time to be in the future.

import time

class App():
    session_length = 10

    def login(self, username):
        self.username = username
        self.start = time.time()

    def access_page(self, username):
        if self.username == username and self.start + self.session_length > time.time():
            return 'approved'
        else:
            return 'expired'


import app
import time


def test_app(monkeypatch):
    user = app.App()
    user.login('foo')
    assert user.access_page('foo') == 'approved'
    current = time.time()
    print(current)

    monkeypatch.setattr(app.time, 'time', lambda : current + 9)
    assert user.access_page('foo') == 'approved'

    monkeypatch.setattr(app.time, 'time', lambda : current + 11)
    assert user.access_page('foo') == 'expired'

Pytest: mocking specific timestamp with datetime

This function will return one of 3 strings based on the date: new_year on January 1st, leap_day on February 29, and regular on every other day. How can we test it?

import datetime

def daily_task():
    now = datetime.datetime.now()
    print(now)
    if now.month == 1 and now.day == 1:
        return 'new_year'
    if now.month == 2 and now.day == 29:
        return 'leap_day'
    return 'regular'

import app

task_name = app.daily_task()
print(task_name)

Pytest: mocking specific timestamp with datetime

import app
import datetime

def test_new_year(monkeypatch):
    mydt = datetime.datetime
    class MyDatetime():
        def now():
            return mydt(2000, 1, 1)

    monkeypatch.setattr(app.datetime, 'datetime', MyDatetime)
    task_name = app.daily_task()
    print(task_name)
    assert task_name == 'new_year'


def test_leap_year(monkeypatch):
    mydt = datetime.datetime
    class MyDatetime():
        def now():
            return mydt(2004, 2, 29)

    monkeypatch.setattr(app.datetime, 'datetime', MyDatetime)
    task_name = app.daily_task()
    print(task_name)
    assert task_name == 'leap_day'


def test_today(monkeypatch):
    mydt = datetime.datetime
    class MyDatetime():
        def now():
            return mydt(2004, 2, 28)

    monkeypatch.setattr(app.datetime, 'datetime', MyDatetime)
    task_name = app.daily_task()
    print(task_name)
    assert task_name == 'regular'

Pytest: mocking datetime.date.today

The datetime class has other methods to retrieve the date (and I could not find how to mock the function deep inside).

import datetime

def get_today():
    return datetime.date.today()


import app

today = app.get_today()
print(type(today))
print(today)

import app
import datetime

def test_new_year(monkeypatch):
    mydt = datetime.date
    class MyDate():
        def today():
            return mydt(2000, 1, 1)

    monkeypatch.setattr(app.datetime, 'date', MyDate)
    today = app.get_today()
    #print(today)
    assert str(today) == '2000-01-01'

def test_leap_year(monkeypatch):
    mydt = datetime.date
    class MyDate():
        def today():
            return mydt(2004, 2, 29)

    monkeypatch.setattr(app.datetime, 'date', MyDate)
    today = app.get_today()
    #print(today)
    assert str(today) == '2004-02-29'

Pytest: mocking datetime date

import datetime
import math

def get_years_passed_category(date_string):
    date = datetime.date.fromisoformat(date_string)
    time_passed = datetime.date.today() - date
    years_passed = time_passed.days // 365
    years_passed_start_and_end_range_tuples_to_category = {
        (0, 1):    "Less than 1 year",
        (1, 5):    "1 - 5 years",
        (5, 10):   "5 - 10 years",
        (10, 20):  "10 - 20 years",
        (20, 30):  "20 - 30 years",
        (30, math.inf): "More than 30 years"
    }
    for (start, end), category in years_passed_start_and_end_range_tuples_to_category.items():
        if start <= years_passed < end:
            return category

    raise ValueError(f"Could not find a years_passed_category for '{date_string}'")

import app

for date in ['2000-01-01', '1990-06-02', '2020-01-01']:
    cat = app.get_years_passed_category(date)
    print(f"{date} : {cat}")


import app
import datetime
import pytest

def test_app(monkeypatch):
    mydt = datetime.date
    class MyDate():
        def today():
            return mydt(2021, 2, 15)
        def fromisoformat(date_str):
            return mydt.fromisoformat(date_str)

    monkeypatch.setattr(app.datetime, 'date', MyDate)

    assert app.get_years_passed_category('1990-06-02') == 'More than 30 years'
    assert app.get_years_passed_category('2000-01-01') == '20 - 30 years'
    assert app.get_years_passed_category('2011-01-01') == '10 - 20 years'
    assert app.get_years_passed_category('2016-01-01') == '5 - 10 years'
    assert app.get_years_passed_category('2020-01-01') == '1 - 5 years'
    assert app.get_years_passed_category('2021-02-14') == 'Less than 1 year'
    assert app.get_years_passed_category('2021-02-15') == 'Less than 1 year'


    with pytest.raises(Exception) as err:
        app.get_years_passed_category('2021-02-16')
    assert err.type == ValueError
    assert str(err.value) == "Could not find a years_passed_category for '2021-02-16'"

Pytest: Mocking environment variables

Just some simple application that uses environment variables.

import os

def add():
    a = os.environ.get("INPUT_A")
    b = os.environ.get("INPUT_B")
    if a is None:
        raise Exception("missing INPUT_A")
    if b is None:
        raise Exception("missing INPUT_B")

    a = int(a)
    b = int(b)
    return a + b

Use the module on the command line:

$ INPUT_A=23 INPUT_B=19 python use_app.py
42

import app

result = app.add()
print(result)

Test the module setting the envrionment variable.

import app
import pytest

def test_app_1(monkeypatch):
    monkeypatch.setenv('INPUT_A', '19')
    monkeypatch.setenv('INPUT_B', '23')
    result = app.add()
    assert result == 42


# We can also test the cases when the environment variables
# are not set or only some of them are set.
def test_app_no_a(monkeypatch):
    with pytest.raises(Exception) as err:
        app.add()
    assert str(err.value) == 'missing INPUT_A'

def test_app_no_b(monkeypatch):
    monkeypatch.setenv('INPUT_A', '0')
    with pytest.raises(Exception) as err:
        app.add()
    assert str(err.value) == 'missing INPUT_B'

Pytest: Mocking The PATH environment variable

import subprocess

def get_python_version():
    proc = subprocess.Popen(['python', '-V'],
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
    )

    out,err = proc.communicate()
    if proc.returncode:
        raise Exception(f"Error exit {proc.returncode}")
    #if err:
    #    raise Exception(f"Error {err}")
    if out:
        return out.decode('utf8') # In Python 3.8.6
    else:
        return err.decode('utf8') # In Python 2.7.18

import app
import os

def test_python():
    out = app.get_python_version()
    assert out == 'Python 3.8.6\n'

def test_in_path(monkeypatch):
    monkeypatch.setenv('PATH', '/usr/bin')
    out = app.get_python_version()
    assert out == 'Python 2.7.18\n'
    print(os.environ['PATH'])
    print()

def test_other():
    print(os.environ['PATH'])
    print()

def test_keep(monkeypatch):
    monkeypatch.setenv('PATH', '/usr/bin' + os.pathsep + os.environ['PATH'])
    print(os.environ['PATH'])

Pytest: One dimensional space-fight

import random

def play():
    debug = False
    move = False
    while True:
        print("\nWelcome to another Number Guessing game")
        hidden = random.randrange(1, 201)
        while True:
            if debug:
                print("Debug: ", hidden)
    
            if move:
                mv = random.randrange(-2, 3)
                hidden = hidden + mv
    
            user_input = input("Please enter your guess [x|s|d|m|n]: ")
            print(user_input)
    
            if user_input == 'x':
                print("Sad to see you leave early")
                return
    
            if user_input == 's':
                print("The hidden value is ", hidden)
                continue
    
            if user_input == 'd':
                debug = not debug
                continue
    
            if user_input == 'm':
                move = not move
                continue
    
            if user_input == 'n':
                print("Giving up, eh?")
                break
    
            guess = int(user_input)
            if guess == hidden:
                print("Hit!")
                break
    
            if guess < hidden:
                print("Your guess is too low")
            else:
                print("Your guess is too high")
    

if __name__ == '__main__':
    play()

Pytest: Mocking input and output in the game

import game
import sys
import io

def test_immediate_exit(monkeypatch, capsys):
    monkeypatch.setattr(sys, 'stdin', io.StringIO('x'))

    game.play()
    out, err = capsys.readouterr()
    assert err == ''

    expected = '''
Welcome to another Number Guessing game
Please enter your guess [x|s|d|m|n]: x
Sad to see you leave early
'''
    assert out == expected

Pytest: Mocking input and output in the game - no tools

import game

def test_immediate_exit():
    input_values = ['x']
    output = []

    def mock_input(s):
       output.append(s)
       return input_values.pop(0)
    game.input = mock_input
    game.print = lambda s : output.append(s)

    game.play()

    assert output == [
        '\nWelcome to another Number Guessing game',
        'Please enter your guess [x|s|d|m|n]: ',
        'x',
        'Sad to see you leave early',
    ]

Pytest: Mocking random in the game

import game
import random
import sys
import io

def test_immediate_exit(monkeypatch, capsys):
    input_values = '\n'.join(['30', '50', '42', 'x'])
    monkeypatch.setattr(sys, 'stdin', io.StringIO(input_values))

    monkeypatch.setattr(random, 'randrange', lambda a, b : 42)

    game.play()
    out, err = capsys.readouterr()

    assert out == '''
Welcome to another Number Guessing game
Please enter your guess [x|s|d|m|n]: 30
Your guess is too low
Please enter your guess [x|s|d|m|n]: 50
Your guess is too high
Please enter your guess [x|s|d|m|n]: 42
Hit!

Welcome to another Number Guessing game
Please enter your guess [x|s|d|m|n]: x
Sad to see you leave early
'''

Pytest: Mocking random in the game - no tools

import game
import random

def test_immediate_exit():
    input_values = ['30', '50', '42', 'x']
    output = []

    def mock_input(s):
       output.append(s)
       return input_values.pop(0)
    game.input = mock_input
    game.print = lambda s : output.append(s)
    random.randrange = lambda a, b : 42

    game.play()

    assert output == [
        '\nWelcome to another Number Guessing game',
        'Please enter your guess [x|s|d|m|n]: ',
        '30',
        'Your guess is too low',
        'Please enter your guess [x|s|d|m|n]: ',
        '50',
        'Your guess is too high',
        'Please enter your guess [x|s|d|m|n]: ',
        '42',
        'Hit!',
        '\nWelcome to another Number Guessing game',
        'Please enter your guess [x|s|d|m|n]: ',
        'x',
        'Sad to see you leave early',
    ]

TODO Pytest: Mocking - collecting stats example

import requests

# An application that allows us to monitor keyword frequency on some popular websites.
# The process:
#    - get the URLs from the database
#    - fetch the content of esch page
#    - get the frequency of keywords for each page
#    - get the precious values from the database
#    - update the database with the new values
#    - send e-mail reporting the changes.

def get_urls():
    #raise Excepton('accessing the database')
    return ['https://code-maven.com/']

def get_content(url, depth):
    #raise Exception(f'donwload content from {url}')
    return "Python Python Pytest Monkey patch Python"

def get_stats(text, limit=None):
    #raise Exception('getting stats from some text')
    return {}

def get_stats_from_db(url):
    #raise Exception('getting stats from database')
    return {}

def create_report(old, new):
    #raise Exception('create report')
    return ''

def send_report(report, subject, to):
    #raise Exception(f'send report to {to}')
    return ''

def main():
    depth = 3
    limit = 17
    boss = 'boss@code-maven.com'
    subject = 'Updated stats'
    urls = get_urls()
    for url in urls:
        content = get_content(url, depth)
        new_stats = get_stats(content, limit)
        old_stats = get_stats_from_db(url)
        report = create_report(old_stats, new_stats)
        send_report(report, subject, boss)

if __name__ == '__main__':
    main()

Flask

We have some examples for the Flask web framework.

You can also check out the full Flask course

Pytest: Flask echo GET

A simple web application with a static page showing an HTML form with a text box and a button. Pressing the button will send the text we enetered in the box to the server which will echo it back.

We need to install flask using pip install flask then we can run the application:

flask --app echo_get --debug run

Then visit http://localhost:5000/ and try the “application”.

Stop the web server by clicking “Ctrl-C”.

from flask import Flask, request

app = Flask(__name__)

@app.route("/")
def hello():
    return '''
<form action="/echo" method="GET">
<input name="text">
<input type="submit" value="Echo">
</form>
'''

@app.route("/echo")
def echo():
    answer = request.args.get('text')
    if answer:
        return "You said: " + answer
    else:
        return "Nothing to say?"

Pytest: testing Flask echo GET

We have a fixture that will return the Flask test_client that we can use in every test function to send a get request.

import echo_get
import pytest


@pytest.fixture()
def app():
    print("setup")
    return echo_get.app.test_client()

def test_main(app):
    rv = app.get('/')
    assert rv.status == '200 OK'
    assert b'<form action="/echo" method="GET">' in rv.data

def test_echo(app):
    rv = app.get('/echo?text=Hello')
    assert rv.status == '200 OK'
    assert b'You said: Hello' in rv.data

def test_empty_echo(app):
    rv = app.get('/echo')
    assert rv.status == '200 OK'
    assert b'Nothing to say?' in rv.data

def test_missing_page(app):
    rv = app.get('/qqrq')
    assert rv.status == '404 NOT FOUND'
    assert b'The requested URL was not found on the server.' in rv.data

Pytest: Flask echo POST

This is very similar to the previous “application” except here the form sends a “POST” request and the server handles that.

from flask import Flask, request

app = Flask(__name__)

@app.route("/")
def main():
    return '''
     <form action="/echo" method="POST">
         <input name="text">
         <input type="submit" value="Echo">
     </form>
     '''

@app.route("/echo", methods=['POST'])
def echo():
    user_text = request.form.get('text', '')
    if user_text:
        return "You said: " + user_text
    return "Nothing to say?"

Run the application

flask --app echo_post --debug run

Visit http://localhost:5000/

Pytest: testing Flask echo POST

Here we can observe how to send data to a “POST” request in a test.

In this case we used a class to write the test. I’d probably prefer the procedural tests as we saw in the previous example, but I wanted to show this as well.

We create a new attribute called “flapp” for the class, something you would not be able to do it stricter languages such as Java. We can only hope that the existing object does not have such an attribute and we are not ruining the class.

import echo_post

class TestEcho:
    def setup_method(self):
        self.flapp = echo_post.app.test_client()
        print("setup")

    def test_main_page(self):
        rv = self.flapp.get('/')
        assert rv.status == '200 OK'
        assert '<form action="/echo" method="POST">' in rv.data.decode('utf-8')

    def test_echo_get(self):
        rv = self.flapp.get('/echo')
        assert rv.status == '405 METHOD NOT ALLOWED'
        assert '<title>405 Method Not Allowed</title>' in rv.data.decode('utf-8')

    def test_echo_post_empty(self):
        rv = self.flapp.post('/echo')
        assert rv.status == '200 OK'
        assert b"Nothing to say?" == rv.data

    def test_echo_post_with_data(self):
        rv = self.flapp.post('/echo', data={ "text": "foo bar" })
        assert rv.status == '200 OK'
        assert b"You said: foo bar" == rv.data

    def test_echo_404(self):
        rv = self.flapp.get('/qqrq')
        assert rv.status == '404 NOT FOUND'
        assert b'The requested URL was not found on the server.' in rv.data

Pytest: Flask app sending mail

This is a web application that allows the user to register by providing an email. The system then generates a unique code and sends an email to the registered address with a link back to the server.

The user is expected to click on the link in the email to verify that it is indeed his email address and he wanted to subscribe.

from flask import Flask, request
import random

app = Flask(__name__)
db = {}

@app.route('/', methods=['GET'])
def home():
    return '''
          <form method="POST" action="/register">
          <input name="email">
          <input type="submit">
          </form>
          '''

@app.route('/register', methods=['POST'])
def register():
    email = request.form.get('email')
    code = str(random.random())
    if db_save(email, code):
        html = '<a href="/verify/{email}/{code}">here</a>'.format(email=email, code = code)
        sendmail({'to': email, 'subject': 'Registration', 'html': html })
        return 'OK'
    else:
        return 'FAILED'

@app.route('/verify/<email>/<code>', methods=['GET'])
def verify(email, code):
    if db_verify(email, code):
        sendmail({'to': email, 'subject': 'Welcome!', 'html': '' })
        return 'OK'
    else:
        return 'FAILED'

def sendmail(data):
    pass

def db_save(email, code):
   if email in db:
       return False
   db[email] = code
   return True

def db_verify(email, code):
    return email in db and db[email] == code

Pytest: Mocking Flask app sending mail

import app
import re

def test_main_page():
    aut = app.app.test_client()

    rv = aut.get('/')
    assert rv.status == '200 OK'
    assert '<form' in str(rv.data)
    assert not 'Welcome back!' in str(rv.data)


def test_verification(monkeypatch):
    aut = app.app.test_client()

    email = 'foo@example.com'

    messages = []
    monkeypatch.setattr('app.sendmail', lambda params: messages.append(params) )

    rv = aut.post('/register', data=dict(email = email ))
    assert rv.status == '200 OK'
    assert 'OK' in str(rv.data)
    print(messages)
    # [{'to': 'foo@example.com', 'subject': 'Registration', 'html': '<a href="/verify/python@example.com/0.81280014">here</a>'}]

    # Remove the html part that we will verify and use later
    html = messages[0].pop('html')

    # Check that the rest of the email is correct
    assert messages == [{'to': 'foo@example.com', 'subject': 'Registration'}]

    # This is the code that we would have received in the email:
    match = re.search(r'/(\d\.\d+)"', html)
    if match:
        code = match.group(1)
    print(code)

    # After the successful verification another email is sent.
    messages = []
    rv = aut.get('/verify/{email}/{code}'.format(email = email, code = code ))
    assert rv.status == '200 OK'
    assert 'OK' in str(rv.data)

    assert messages == [{'to': email, 'subject': 'Welcome!', 'html': ''}]

def test_invalid_verification(monkeypatch):
    aut = app.app.test_client()

    email = 'bar@example.com'

    messages = []
    monkeypatch.setattr('app.sendmail', lambda params: messages.append(params) )

    rv = aut.post('/register', data=dict(email = email ))
    assert rv.status == '200 OK'
    assert 'OK' in str(rv.data)

    messages = []
    # Test what happens if we use an incorrect code to verify the email address:
    rv = aut.get('/verify/{email}/{code}'.format(email = email, code = 'other' ))
    assert rv.status == '200 OK'
    assert 'FAILED' in str(rv.data)

    # No email was sent
    assert messages == []

Testing Flask in Docker

We create a small Flask-based web application.

  1. First we test it internally using the test_client feature flask provied.
  2. Then we start the web application and test it with external tools.

We could start the web application natively on our operating system, but for more complex applications it will be probably a good idea to use a container to make it repeatable.

So we create a Docker image and run the application in a Docker container.

Actually we let the test build both the image and run the container.

Pytest with Docker - application

This is the Flask-based web application.

The main page returns some text.

The ’/api/calcURL accepts 2 parameters:aandb` with two numbers. It returns a JSON with these values and their sum.

from flask import Flask, jsonify, request
import time
calcapp = Flask(__name__)

@calcapp.route("/")
def main():
    return 'Send a GET request to /api/calc and get a JSON response.'

@calcapp.route("/api/calc")
def add():
    a = int(request.args.get('a', 0))
    b = int(request.args.get('b', 0))
    return jsonify({
        "a"        :  a,
        "b"        :  b,
        "add"      :  a+b,
    })

Pytest with Docker - test internally

This is the same type of test we aready saw in which we use the test_client of Flask and we do not start a real web server.

pytest test_app.py

import app


def test_app():
    web = app.calcapp.test_client()

    rv = web.get('/')
    assert rv.status == '200 OK'
    assert b'Send a GET request to /api/calc and get a JSON response.' == rv.data

def test_calc():
    web = app.calcapp.test_client()

    rv = web.get('/api/calc?a=10&b=2')
    assert rv.status == '200 OK'
    assert rv.headers['Content-Type'] == 'application/json'
    resp = rv.json
    assert resp == {
        "a"        :  10,
        "b"        :  2,
        "add"      :  12,
    }

Docker to run the Flask app

A simple Dockerfile to create a Docker image. You might need a lot more complex one for your application, but the general usage will be the same.

FROM python:3.14
WORKDIR /workdir
RUN pip install flask
CMD ["flask", "run", "--host", "0.0.0.0"]

We also have a .dockerignore file to make the building of the image faster.

__pycache__
.git
.pytest_cache

We can (and probably should) build the image manually with the following command. This will speed up the tests as the test will be able to reuse this image. The name of the image does not really matter here.

$ docker build myimg -t .

We can run the container manually

docker run --rm -v$(pwd):/workdir -p5001:5000 myimg

Then we can access the web site from our computer using this address:

http://localhost:5001

Pytest with Docker - test

This test uses xUnit-style fixtures and because we need to pass around some values it uses some global variable. Not ideal.

The fixtures build the image and run the container that launches the web application.

Then the tests use request to access the site.

import requests
import time
import os

def setup_module():
    global image_name
    image_name = 'test_image_' + str(int(time.time()*1000))
    print(f"image: {image_name}")
    print("setup_module ", os.system(f"docker build -t {image_name} ."))

def teardown_module():
    print("teardown_module ", os.system(f"docker rmi -f {image_name}"))


def setup_function():
    global port
    global container_name

    port = '5001'
    container_name = 'test_container_' + str(int(time.time()*1000))
    print(f"container: {container_name}")
    print("setup_function ", os.system(f"docker run --rm -d -v$(pwd):/workdir -p{port}:5000 --name {container_name} {image_name}"))
    time.sleep(1) # Let the Docker container start

def teardown_function():
    print("teardown_function ", os.system(f"docker stop -t 0 {container_name}"))

def test_app():
    url = f"http://localhost:{port}/api/calc?a=3&b=10"
    print(url)
    res = requests.get(url)
    assert res.status_code == 200
    assert res.json() == {'a': 3, 'add': 13, 'b': 10}

pytest test_with_docker.py

Pytest with Docker - improved

import pytest
import requests
import time
import os

#@pytest.fixture(autouse = True, scope="module")
@pytest.fixture()
def image():
    image_name = 'test_image_' + str(int(time.time()*1000))
    print(f"image: {image_name}")
    print("setup_module ", os.system(f"docker build -t {image_name} ."))

    yield image_name

    print("teardown_module ", os.system(f"docker rmi -f {image_name}"))


@pytest.fixture()
def myport(image):
    port = '5001'
    container_name = 'test_container_' + str(int(time.time()*1000))
    print(f"container: {container_name}")
    print("setup_function ", os.system(f"docker run --rm -d -v$(pwd):/workdir -p{port}:5000 --name {container_name} {image}"))
    time.sleep(1) # Let the Docker container start

    yield port

    print("teardown_function ", os.system(f"docker stop -t 0 {container_name}"))

def test_app(myport):
    url = f"http://localhost:{myport}/api/calc?a=3&b=10"
    print(url)
    res = requests.get(url)
    assert res.status_code == 200
    assert res.json() == {'a': 3, 'add': 13, 'b': 10}

def test_app_again(myport):
    url = f"http://localhost:{myport}/api/calc?a=-1&b=1"
    print(url)
    res = requests.get(url)
    assert res.status_code == 200
    assert res.json() == {'a': -1, 'add': 0, 'b': 1}

Pytest command line options

PyTest: Run tests in parallel with xdist

By default pytest runs the test-unctions sequentially. So if we have 4 test-function each of them taking 2 seconds, the whole test-suite will run 8 seconds.

The pytest-xdist plugin adds a new command-line flag -n that allows us to set the number of workers. This makes it possible to run tests in parallel.

For this it is extremly important that the tests are indpendent and they don’t use the same resource. (eg. each one will use its own folder or its own database)

$ pip install pytest-xdist
$ pytest -n NUM

We have two test files, each of them two tests taking 2 seconds each:

import time

def test_dog():
    time.sleep(2)

def test_cat():
    time.sleep(2)


import time

def test_blue():
    time.sleep(2)

def test_green():
    time.sleep(2)


$ time pytest          8.04 sec
$ time pytest -n 2     4.64 sec
$ time pytest -n 4     3.07 sec

$ time pytest -n auto  3.96 sec using 8 workers

PyTest: Order of tests

Pytest runs the test in the same order as they are found in the test module:

def test_one():
    assert True

def test_two():
    assert True

def test_three():
    assert True


pytest -v test_order.py

test_order.py::test_one PASSED
test_order.py::test_two PASSED
test_order.py::test_three PASSED

PyTest: Randomize Order of tests

The pytest-random-order provides a new command-line flag --random-order that will randomize the test-function.

pip install pytest-random-order

You might need to run the tests several times till you see exactly this random order.

$ pytest -v --random-order test_order.py

test_order.py::test_two PASSED
test_order.py::test_three PASSED
test_order.py::test_one PASSED

PyTest: Force default order

If for some reason we would like to make sure the order remains the same in a given file even when using the --random-order flag, we can add the following two lines of code.

import pytest
pytestmark = pytest.mark.random_order(disabled=True)

import pytest
pytestmark = pytest.mark.random_order(disabled=True)

def test_one():
    assert True

def test_two():
    assert True

def test_three():
    assert True



$ pytest -v --random-order

test_order.py::test_three PASSED
test_order.py::test_one PASSED
test_order.py::test_two PASSED
test_default_order.py::test_one PASSED
test_default_order.py::test_two PASSED
test_default_order.py::test_three PASSED

PyTest test discovery

Running pytest will find test files and in the files test functions.

  • test_*.py files
  • *_test.py files
  • test_* functions
  • TestSomething class
  • test_* methods
examples/pytest/discovery
.
├── db
│   └── test_db.py
├── other_file.py
├── test_one.py
└── two_test.py


============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /home/gabor/venv3/bin/python
cachedir: .pytest_cache
plugins: json-report-1.2.4, random-order-1.0.4, flake8-1.0.6, forked-1.3.0, dash-1.17.0, metadata-1.11.0, xdist-2.2.1
collecting ... collected 3 items

test_one.py::test_1 PASSED                                               [ 33%]
two_test.py::test_2 PASSED                                               [ 66%]
db/test_db.py::test_db PASSED                                            [100%]

============================== 3 passed in 0.02s ===============================

Pytest dry-run --collect-only

  • Find all the test files, test classes, test functions that will be executed.
  • But don’t run them…
  • … but they are still loaded into memory so any code in the “body” of the files is executed.
pytest --collect-only

PyTest test discovery - ignore some tests

$ pytest


$ pytest --ignore venv3/

Ignore the tests in the venv3 folder.

test_mymod_1.py .
test_mymod_2.py .F

PyTest select tests by name -k

  • –collect-only - only list the tests, don’t run them yet.
  • -k select by name

def test_database_read():
    assert True

def test_database_write():
    assert True

def test_database_forget():
    assert True

def test_ui_access():
    assert True

def test_ui_forget():
    assert True

pytest --collect-only test_by_name.py
    test_database_read
    test_database_write
    test_database_forget
    test_ui_access
    test_ui_forget

pytest --collect-only -k database test_by_name.py
    test_database_forget
    test_database_read
    test_database_write

pytest --collect-only -k ui test_by_name.py
    test_ui_access
    test_ui_forget

pytest --collect-only -k forget test_by_name.py
    test_database_forget
    test_ui_forget

pytest --collect-only -k "forget or read" test_by_name.py
    test_database_read
    test_database_forget
    test_ui_forget

Pytest use markers and select tests using -m

  • Use the @pytest.mark.name decorator to tag the tests.
import pytest

@pytest.mark.smoke
def test_database_read():
    assert True

@pytest.mark.security
@pytest.mark.smoke
def test_database_write():
    assert True

@pytest.mark.security
def test_database_forget():
    assert True

@pytest.mark.smoke
def test_ui_access():
    assert True

@pytest.mark.security
def test_ui_forget():
    assert True


pytest --collect-only -m security test_by_marker.py
    test_ui_forget
    test_database_write
    test_database_forget

pytest --collect-only -m smoke test_by_marker.py
    test_database_read
    test_ui_access
    test_database_write
  • We need to declare them in the pytest.ini to avoid the warning
[pytest]
markers =
    smoke: Smoke tests
    security: DevSecOps
    # long: Some long running tests

No test selected

If you run pytest and it cannot find any tests, for example because you used some selector and no test matched it, then Pytest will exit with exit code 5.

This is considered a failure by every tool, including Jenkins and other CI systems.

On the other hand you won’t see any failed test reported. After all if no tests are run, then none of them fails. This can be confusing. “CI failed, but no test failed and nothing is reported.”

$ pytest -k long test_by_marker.py


============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest, configfile: pytest.ini
plugins: flake8-1.0.6, dash-1.17.0
collected 5 items / 5 deselected

============================ 5 deselected in 0.00s =============================

$ echo $?
5

Pytest reporting in JUnit, XML, or JSON format

Here we have some tests with various special marks we have seen earlier.

import pytest


def test_blue():
    pass


def test_red():
    assert 1 == 2


@pytest.mark.skip(reason="So we can show skip reporting")
def test_purple():
    assert 1 == 3


@pytest.mark.xfail(reason="To show xfail that really fails")
def test_orange():
    1 == 4


@pytest.mark.xfail(reason="To show xfail that passes")
def test_green():
    pass

Pytest reporting in JUnit XML format

  • e.g. for Jenkins integration
  • See usage
pytest --junitxml report.xml

<?xml version="1.0" encoding="utf-8"?><testsuites name="pytest tests"><testsuite name="pytest" errors="0" failures="1" skipped="1" tests="5" time="0.128" timestamp="2026-03-19T16:57:19.305310+01:00" hostname="code-maven"><testcase classname="books.python-testing.src.examples.pytest.reporting.test_colors" name="test_blue" time="0.001" /><testcase classname="books.python-testing.src.examples.pytest.reporting.test_colors" name="test_red" time="0.001"><failure message="assert 1 == 2">def test_red():
&gt;       assert 1 == 2
E       assert 1 == 2

test_colors.py:7: AssertionError</failure></testcase><testcase classname="books.python-testing.src.examples.pytest.reporting.test_colors" name="test_purple" time="0.000"><skipped type="pytest.skip" message="So we can show skip reporting">/home/gabor/github/code-maven.com/python.code-maven.com/books/python-testing/src/examples/pytest/reporting/test_colors.py:9: So we can show skip reporting</skipped></testcase><testcase classname="books.python-testing.src.examples.pytest.reporting.test_colors" name="test_orange" time="0.000" /><testcase classname="books.python-testing.src.examples.pytest.reporting.test_colors" name="test_green" time="0.000" /></testsuite></testsuites>

To make the XML more human-readable:

cat report.xml | python -c 'import sys;import xml.dom.minidom;s=sys.stdin.read();print(xml.dom.minidom.parseString(s).toprettyxml())'

Pytest reporting in JSON format

pip install pytest-json-report

pytest --json-report --json-report-file=report.json --json-report-indent=4

Recommended to also add

--json-report-omit=log

pytest -s --json-report --json-report-file=report.json --log-cli-level=INFO

Pytest JSON report

{
    "created": 1773935816.731533,
    "duration": 0.12311553955078125,
    "exitcode": 1,
    "root": "/home/gabor/github/code-maven.com/python.code-maven.com",
    "environment": {},
    "summary": {
        "passed": 1,
        "failed": 1,
        "skipped": 1,
        "xpassed": 2,
        "total": 5,
        "collected": 5
    },
    "collectors": [
        {
            "nodeid": "",
            "outcome": "passed",
            "result": [
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting",
                    "type": "Dir"
                }
            ]
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py",
            "outcome": "passed",
            "result": [
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_blue",
                    "type": "Function",
                    "lineno": 2
                },
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_red",
                    "type": "Function",
                    "lineno": 5
                },
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_purple",
                    "type": "Function",
                    "lineno": 8
                },
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_orange",
                    "type": "Function",
                    "lineno": 12
                },
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_green",
                    "type": "Function",
                    "lineno": 16
                }
            ]
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting",
            "outcome": "passed",
            "result": [
                {
                    "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py",
                    "type": "Module"
                }
            ]
        }
    ],
    "tests": [
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_blue",
            "lineno": 2,
            "outcome": "passed",
            "keywords": [
                "test_blue",
                "test_colors.py",
                "reporting",
                "pytest",
                "examples",
                "src",
                "python-testing",
                "books",
                "python.code-maven.com",
                ""
            ],
            "setup": {
                "duration": 0.0002916679950430989,
                "outcome": "passed"
            },
            "call": {
                "duration": 0.00021083600586280227,
                "outcome": "passed"
            },
            "teardown": {
                "duration": 0.00015628400433342904,
                "outcome": "passed"
            }
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_red",
            "lineno": 5,
            "outcome": "failed",
            "keywords": [
                "test_red",
                "test_colors.py",
                "reporting",
                "pytest",
                "examples",
                "src",
                "python-testing",
                "books",
                "python.code-maven.com",
                ""
            ],
            "setup": {
                "duration": 0.00014479199307970703,
                "outcome": "passed"
            },
            "call": {
                "duration": 0.00047524301044177264,
                "outcome": "failed",
                "crash": {
                    "path": "/home/gabor/github/code-maven.com/python.code-maven.com/books/python-testing/src/examples/pytest/reporting/test_colors.py",
                    "lineno": 7,
                    "message": "assert 1 == 2"
                },
                "traceback": [
                    {
                        "path": "test_colors.py",
                        "lineno": 7,
                        "message": "AssertionError"
                    }
                ],
                "longrepr": "def test_red():\n>       assert 1 == 2\nE       assert 1 == 2\n\ntest_colors.py:7: AssertionError"
            },
            "teardown": {
                "duration": 0.0003812770009972155,
                "outcome": "passed"
            }
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_purple",
            "lineno": 8,
            "outcome": "skipped",
            "keywords": [
                "test_purple",
                "skip",
                "pytestmark",
                "test_colors.py",
                "reporting",
                "pytest",
                "examples",
                "src",
                "python-testing",
                "books",
                "python.code-maven.com",
                ""
            ],
            "setup": {
                "duration": 0.00021005500457249582,
                "outcome": "skipped",
                "longrepr": "('/home/gabor/github/code-maven.com/python.code-maven.com/books/python-testing/src/examples/pytest/reporting/test_colors.py', 9, 'Skipped: So we can show skip reporting')"
            },
            "teardown": {
                "duration": 0.0001474780001444742,
                "outcome": "passed"
            }
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_orange",
            "lineno": 12,
            "outcome": "xpassed",
            "keywords": [
                "test_orange",
                "xfail",
                "pytestmark",
                "test_colors.py",
                "reporting",
                "pytest",
                "examples",
                "src",
                "python-testing",
                "books",
                "python.code-maven.com",
                ""
            ],
            "setup": {
                "duration": 0.00017399700300302356,
                "outcome": "passed"
            },
            "call": {
                "duration": 0.0001693279918981716,
                "outcome": "passed"
            },
            "teardown": {
                "duration": 0.00013180699897930026,
                "outcome": "passed"
            }
        },
        {
            "nodeid": "books/python-testing/src/examples/pytest/reporting/test_colors.py::test_green",
            "lineno": 16,
            "outcome": "xpassed",
            "keywords": [
                "test_green",
                "xfail",
                "pytestmark",
                "test_colors.py",
                "reporting",
                "pytest",
                "examples",
                "src",
                "python-testing",
                "books",
                "python.code-maven.com",
                ""
            ],
            "setup": {
                "duration": 0.00013885101361665875,
                "outcome": "passed"
            },
            "call": {
                "duration": 0.00016100300126709044,
                "outcome": "passed"
            },
            "teardown": {
                "duration": 0.00013386199134401977,
                "outcome": "passed"
            }
        }
    ]
}

Add extra command line parameters to Pytest - conftest - getoption

In the conftest.py file we can use the pytest_addoption function to defined new command line options.

  • --demo expects a value.
  • --noisy is a boolean flag. By default it is false.

In the tests we need to use the request fixture and the getoption method to get the value of each option

def pytest_addoption(parser):
    parser.addoption("--demo")
    parser.addoption("--noisy", action='store_true')

def test_me(request):
    print(request.config.getoption("--demo"))
    print(request.config.getoption("--noisy"))


pytest -s
None
False

pytest -s --demo Hello --noisy
Hello
True

Add extra command line parameters to Pytest - as a fixture

  • conftest

  • We can also create a fixture that will read the parameter

import pytest

def pytest_addoption(parser):
    parser.addoption("--demo")

@pytest.fixture
def mydemo(request):
    return request.config.getoption("--demo")

def test_me(mydemo):
    print(mydemo)

pytest -s
test_one.py None

pytest -s --demo Hello
test_one.py Hello

Add extra command line parameters to Pytest - used in the autouse fixtures

  • conftest
#import pytest
#
def pytest_addoption(parser):
    parser.addoption("--demo")
#
#@pytest.fixture
#def demo(request):
#    return request.config.getoption("--demo")

import pytest

@pytest.fixture(autouse = True, scope="module")
def module_demo(request):
    demo = request.config.getoption("--demo")
    print(f"Module {demo}")
    return demo


@pytest.fixture(autouse = True, scope="function")
def func_demo(request):
    demo = request.config.getoption("--demo")
    print(f"Func {demo}")
    return demo


def test_me():
    pass

def test_two():
    pass

pytest -s --demo Hello

Module Hello
Func Hello
Func Hello

PyTest: Test Coverage

def fib(n):
    if n < 1:
        raise ValueError(f'Invalid parameter was given {n}')
    a, b = 1, 1
    for _ in range(1, n):
        a, b = b, a+b
    return a

def add(x, y):
    return x + y

def area(x, y):
    if x > 0 and y > 0:
        return x * y
    else:
        return None

import mymod

def test_add():
    assert mymod.add(2, 3) == 5

def test_area():
    assert mymod.area(2, 3) == 6
    assert mymod.area(-2, 3) == None

def test_fib():
    assert mymod.fib(1) == 1

pip install pytest-cov

pytest --cov=mymod --cov-report html --cov-branch

Open htmlcov/index.html

Static Analyzers (Linters)

There are various static Analyzers for Python. We’ll take a look at a few of them.

Some of these tools will report on stylistic issue (e.g. pus spaces around operators), others will complain about the lack of documentation, but they can also point at potential bugs. For example if the same function name is defined twice, or if there is a function that is being used but never declared.

If you wonder how the latter could happen, wouldn’t the program crash. Well, if the function is rarely called and there are no tests covering the case when it is used then the software might run well for a long time before it will crash at 2 am on a Saturday.

We’ll also see how to integrate them with testing.

Pytest and flake8

flake8 - Style Guide Enforcement

import sys

def add(a):
    return a

def add(x, y):
   z = 42
   sum = x+y
   return sum

print = 42

import mymod

def test_add():
    assert mymod.add(2, 3) == 5


[flake8]
ignore =
#ignore = F401 E302 E111 F841 E111 E226 E111 W391
#exclude = test_mymod.py


pip install flake8
pip install pytest-flake8
pip install flake8-builtins

flake8
rm -rf .pytest_cache/
pytest --flake8

Pytest and pylint

Pylint

pip install pylint

pylint .

Pytest and ruff

ruff

pip install ruff

ruff check

Pytest and mypy

mypy provides (optional) type-checking for Python.

You can annotate your code with type information just like in the strongly typed languages like C or Java. Python will happily let you do that and promptly disregard them.

However they will help you reading the code. They will help your IDE and you can check them using mypy.

It is recommended that you configure mypy to run in both the CI and in the pre-commit hook.

Install

$ pip install mypy
$ pip install pytest-mypy

Run mypy

$ mypy mymod.py

$ pytest --mypy

Sample code

import sys

z = "23"
z = int(z)

def add(x, y):
   return x + y

Regular test

import mymod

x = "23"
x = int(x)

def test_add():
    assert mymod.add(2, 3) == 5

Run the test

pytest

Run mypy

$ mypy mymod.py
mymod.py:4: error: Incompatible types in assignment (expression has type "int", variable has type "str")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Run mypy with pytest

$ pytest --mypy

mymod.py FF
test_mymod.py F.

Excluding files when using mypy works, but that does not exclude them when using pytest --mypy

[mypy]
exclude=test_mymod.py

Not even this:

[mypy]
exclude=test_mymod.py

Black code formatter

The black code formatter.

$ pip install black

In the CI one can use the --check flag. It won’t change the code. It will only report if the code is formatted according to its rules. It will fail if black would want to change the format.

$ black --check .

Read the help and the documentation

black -h

Pytest - other

Testing Master Mind

import random


def main():
    hidden = [str(x) for x in random.sample(range(1, 7), 4)]
    #print(hidden)
    
    while(True):
        print("Please enter 4 digits")
        guess = list(input())
        if len(guess)!=4:
            continue    
        res = ""
        for h, g in zip(hidden, guess):
           #print(h, g)
           if h == g:
              res += "b"
           elif g in hidden:
              res += "w"
           #print(res)
        if res=='bbbb':
           print("Congrats!") 
           break   
        print(''.join(sorted(res)))


if __name__ == "__main__":
    main() 

import random
import master_mind as mm

def test_mm():
    random.sample = lambda a, b: [1,2,3,4]
    input_values = ['1234']
    output = []

    def mock_input():
        #output.append(s)
        return input_values.pop(0)
    mm.input = mock_input
    mm.print = lambda *s : output.append(s)

    mm.main()

    assert output == [
        ("Please enter 4 digits",),
        ('Congrats!',),
    ]


def test_wrong():
    random.sample = lambda a, b: [1,2,3,4]
    input_values = ['1235', '1234']
    output = []

    def mock_input():
        #output.append(s)
        return input_values.pop(0)
    mm.input = mock_input
    mm.print = lambda *s : output.append(s)

    mm.main()

    assert output == [
        ("Please enter 4 digits",),
        ("bbb",),
        ("Please enter 4 digits",),
        ('Congrats!',),
    ]

Module Fibonacci

def fibonacci_number(n):
    if n==1:
        return 1
    if n==2:
        return 1
    if n==3:
        return 5

    return 'unimplemented'

def fibonacci_list(n):
    if n == 1:
        return [1]
    if n == 2:
        return [1, 1]
    if n == 3:
        return [1, 1, 5]
    raise Exception('unimplemented')

PyTest - assertion

import mymath

def test_fibonacci():
    assert mymath.fibonacci(1) == 1


$ py.test test_fibonacci_ok.py
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2
collected 1 items

test_fibonacci_ok.py .

=========================== 1 passed in 0.01 seconds ===========================

PyTest - failure

import mymath

def test_fibonacci():
    assert mymath.fibonacci(1) == 1
    assert mymath.fibonacci(2) == 1
    assert mymath.fibonacci(3) == 2


$ py.test test_fibonacci.py
============================== test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2
collected 1 items 

test_fibonacci.py F

=================================== FAILURES ====================================
________________________________ test_fibonacci _________________________________

    def test_fibonacci():
        assert mymath.fibonacci(1) == 1
        assert mymath.fibonacci(2) == 1
>       assert mymath.fibonacci(3) == 2
E       assert 5 == 2
E        +  where 5 = <function fibonacci at 0x10a024500>(3)
E        +    where <function fibonacci at 0x10a024500> = mymath.fibonacci

test_fibonacci.py:6: AssertionError
=========================== 1 failed in 0.02 seconds ============================

PyTest - list

import fibo

def test_fibonacci_number():
    assert fibo.fibonacci_number(1) == 1
    assert fibo.fibonacci_number(2) == 1
    assert fibo.fibonacci_number(3) == 2
    assert fibo.fibonacci_number(4) == 2

def test_fibo():
    assert fibo.fibonacci_list(1) == [1]
    assert fibo.fibonacci_list(2) == [1, 1]
    assert fibo.fibonacci_list(3) == [1, 1, 2]

$ py.test test_fibo.py 
========================== test session starts ===========================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2
collected 1 items 

test_fibo.py F

================================ FAILURES ================================
_______________________________ test_fibo ________________________________

    def test_fibo():
        assert mymath.fibo(1) == [1]
        assert mymath.fibo(2) == [1, 1]
>       assert mymath.fibo(3) == [1, 1, 2]
E       assert [1, 1, 5] == [1, 1, 2]
E         At index 2 diff: 5 != 2

test_fibo.py:6: AssertionError
======================== 1 failed in 0.01 seconds ========================

Pytest: monkeypatching time

import time

def now():
    return time.time()


import app
import time

def test_one():
    our_real_1 = time.time()
    their_real_1 = app.now()
    assert abs(our_real_1 - their_real_1) < 0.00001

    app.time.time = lambda : our_real_1 + 100

    our_real_2 = time.time()
    print (our_real_2 - our_real_1)
    #their_real_2 = app.now()
    #assert abs(our_real_2 - their_real_2) >= 100



from time import time

def now():
    return time()


import app
import time

def test_one():
    our_real_1 = time.time()
    their_real_1 = app.now()
    assert abs(our_real_1 - their_real_1) < 0.0001

    app.time = lambda : our_real_1 + 100

    our_real_2 = time.time()
    assert abs(our_real_2 - our_real_1) < 0.0001

    their_real_2 = app.now()
    assert abs(our_real_2 - their_real_2) >= 99

def test_two():
    our_real_1 = time.time()
    their_real_1 = app.now()
    assert abs(our_real_1 - their_real_1) < 0.0001





import app
import time

def test_one(monkeypatch):
    our_real_1 = time.time()
    their_real_1 = app.now()
    assert abs(our_real_1 - their_real_1) < 0.0001

    monkeypatch.setattr(app, 'time', lambda : our_real_1 + 100)

    our_real_2 = time.time()
    assert abs(our_real_2 - our_real_1) < 0.0001

    their_real_2 = app.now()
    assert abs(our_real_2 - their_real_2) >= 99

def test_two():
    our_real_1 = time.time()
    their_real_1 = app.now()
    assert abs(our_real_1 - their_real_1) < 0.00001

PyTest: no random order

pytest -p no:random-order -v

#import pytest

def pytest_addoption(parser):
    parser.addoption("--demo")

#@pytest.fixture
#def demo(request):
#    return request.config.getoption("--demo")

import pytest

@pytest.fixture
def mydemo(request):
    demo = request.config.getoption("--demo")
    print(f"In fixture {demo}")
    return demo


def test_me(mydemo):
    print(f"In test {mydemo}")

PyTest bank deposit

class NegativeDeposite(Exception):
    pass

class Bank:
    def __init__(self, start):
        self.balance = start

    def deposit(self, money):
        if money < 0:
            raise NegativeDeposite('Cannot deposit negative sum')
        self.balance += money
        return

PyTest expected exceptions (bank deposit)

import pytest
from banks import Bank, NegativeDeposite


def test_negative_deposit():
    b = Bank(10)
    with pytest.raises(Exception) as exinfo:
        b.deposit(-1)
    assert exinfo.type == NegativeDeposite
    assert str(exinfo.value) == 'Cannot deposit negative sum'

pytest test_bank.py

test_bank.py .

PyTest expected exceptions (bank deposit) - no exception happens

Pytest properly reports that there was no exception where an exception was expected.

class NegativeDeposite(Exception):
    pass

class Bank:
    def __init__(self, start):
        self.balance = start

    def deposit(self, money):
        #if money < 0:
        #    raise NegativeDeposite('Cannot deposit negative sum')
        self.balance += money
        return


    def test_negative_deposit():
        b = Bank(10)
        with pytest.raises(NegativeDeposite) as e:
>           b.deposit(-1)
E           Failed: DID NOT RAISE <class 'Exception'>

PyTest expected exceptions (bank deposit) - different exception is raised

class NegativeDeposite(Exception):
    pass

class Bank:
    def __init__(self, start):
        self.balance = start

    def deposit(self, money):
        if money < 0:
            raise ValueError('Cannot deposit negative sum')
        self.balance += money
        return



    def test_negative_deposit():
        b = Bank(10)
        with pytest.raises(Exception) as exinfo:
            b.deposit(-1)
>       assert exinfo.type == NegativeDeposite
E       AssertionError: assert <class 'ValueError'> == NegativeDeposite
E        +  where <class 'ValueError'> = <ExceptionInfo ValueError tblen=2>.type

PyTest expected exceptions - divide

  • Some older slides I kept them around
import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as err:
        divide(1, 0)
    assert str(err.value) == 'Cannot divide by Zero'

#divide(3, 0)

PyTest expected exceptions output

$ pytest test_exceptions.py

test_exceptions.py .

PyTest expected exceptions (text changed)

import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Null')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero'

PyTest expected exceptions (text changed) output

$ pytest test_exceptions_text_changed.py


    def test_zero_division():
        with pytest.raises(ValueError) as e:
            divide(1, 0)
>       assert str(e.value) == 'Cannot divide by Zero'
E       AssertionError: assert 'Cannot divide by Null' == 'Cannot divide by Zero'
E         - Cannot divide by Null
E         ?                  ^^^^
E         + Cannot divide by Zero
E         ?                  ^^^^

PyTest expected exceptions (other exception)

import pytest

def divide(a, b):
#    if b == 0:
#        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero'

PyTest expected exceptions (other exception) output

    $ pytest test_exceptions_failing.py

    def test_zero_division():
        with pytest.raises(ValueError) as e:
>           divide(1, 0)

test_exceptions_failing.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 1, b = 0

    def divide(a, b):
    #    if b == 0:
    #        raise ValueError('Cannot divide by Zero')
>       return a / b
E       ZeroDivisionError: division by zero

PyTest expected exceptions (no exception)

import pytest

def divide(a, b):
    if b == 0:
        return None
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero'

PyTest expected exceptions (no exception) output

    def test_zero_division():
        with pytest.raises(ValueError) as e:
>           divide(1, 0)
E           Failed: DID NOT RAISE <class 'ValueError'>

Exercise: test this app

Write tests for the swap and average functions of the app module. Can you find a bug?


def swap(txt):
    '''
    >>> half("abcd"))
    cdab
    '''
    return txt[int(len(txt)/2):] + txt[:int(len(txt)/2)]

def average(*numbers):
    '''
    >>> average(2, 4, 6)
    4
    '''
    s = 0
    c = 0
    for n in numbers:
        s += n
        c += 1
    return s/c

Exercise: test the csv module

  • csv
  • Create a CSV file, read it and check if the results are as expected!
  • Test creating a CSV file?
  • Test round trip?

Solution: Pytest test this app

import app

def test_swap():
    assert app.swap("abcd") == "cdab"
    assert app.swap("abc") == "bca"
    assert app.swap("abcde") == "cdeab"
    assert app.swap("a") == "a"
    assert app.swap("") == ""

def test_average():
    assert app.average(2, 4) == 3
    assert app.average(2, 3) == 2.5
    assert app.average(42) == 42
    #assert app.average() == 0

Solution: test the csv module

{% embed include file=“src/examples/csv/csv_file_newline.csv)

import csv


def test_csv():
    filename = '../../examples/csv/process_csv_file_newline.csv'
    with open(filename) as fh:
        rd = csv.reader(fh, delimiter=';')
        assert rd.__next__() == ['Tudor', 'Vidor', '10', 'Hapci']
        assert rd.__next__() == ['Szundi', 'Morgo', '7', 'Szende']
        assert rd.__next__() == ['Kuka', 'Hofeherke; \nalma', '100', 'Kiralyno']
        assert rd.__next__() == ['Boszorkany', 'Herceg', '9', 'Meselo']

PyTest using classes

class TestClass():
    def test_one(self):
        print("one")
        assert True
        print("one after")

    def test_two(self):
        print("two")
        assert False
        print("two after")

class TestBad():
    def test_three(self):
        print("three")
        assert False
        print("three after")

Output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest
plugins: flake8-1.0.6, dash-1.17.0
collected 3 items

test_with_class.py .FF                                                   [100%]

=================================== FAILURES ===================================
______________________________ TestClass.test_two ______________________________

self = <test_with_class.TestClass object at 0x7fac08abdbe0>

    def test_two(self):
        print("two")
>       assert False
E       assert False

test_with_class.py:9: AssertionError
----------------------------- Captured stdout call -----------------------------
two
______________________________ TestBad.test_three ______________________________

self = <test_with_class.TestBad object at 0x7fac08a606a0>

    def test_three(self):
        print("three")
>       assert False
E       assert False

test_with_class.py:15: AssertionError
----------------------------- Captured stdout call -----------------------------
three
=========================== short test summary info ============================
FAILED test_with_class.py::TestClass::test_two - assert False
FAILED test_with_class.py::TestBad::test_three - assert False
========================= 2 failed, 1 passed in 0.03s ==========================

Exercise: module

Pick one of the modules and write a test for it.

Exercise: Open Source

  • Visit the stats on PyDigger.com
  • List the packages that have GitHub no Travis-CI.
  • Pick one that sounds simple. Visit its GitHub page and check if it has tests.
  • If it does not, wirte one.
  • Send Pull Request

Pytest and forking

  • This tests passes and generates two reports.
  • I could not find a way yet to avoid the reporting in the child-process. Maybe we need to run this with a special runner that will fork and run this test on our behalf.
import os

def func(x, y):
    pid = os.fork()
    if pid == 0:
        print(f"Child {os.getpid()}")
        #raise Exception("hello")
        exit()
    print(f"Parent {os.getpid()} The child is {pid}")
    os.wait()
    #exit()
    #raise Exception("hello")
    return x+y
  

if __name__ == '__main__':
    func(2, 3)

import app
import os

#def test_func():
#   assert app.func(2, 3) == 5


def test_func():
    pid = os.getpid()
    try:
        res = app.func(2, 3)
        assert res == 5
    except SystemExit as ex:
        assert str(ex) == 'None'
        #SystemExit(None)
        # or ex == 0
        if pid == os.getpid():
            raise ex


import logging

def add(x, y):
#    logger = logging.getLogger("mytest")
    logging.basicConfig(level = logging.INFO)
    logging.info("Just some info log")
    return x * y

def test_one():
    assert add(2, 2) == 4

Exercise: Write tests for script combining files

  • This is a solution for one of the exercises in which we had to combine two files adding the numbers of the vegetables together.
  • Many things could be improved, but before doing that, write a test (or two) to check this code. Without changing it.
c = {}
with open('examples/files/a.txt') as fh:
    for line in fh:
        k, v = line.rstrip("\n").split("=")
        if k in c:
            c[k] += int(v)
        else:
            c[k] = int(v)

with open('examples/files/b.txt') as fh:
    for line in fh:
        k, v = line.rstrip("\n").split("=")
        if k in c:
            c[k] += int(v)
        else:
            c[k] = int(v)


with open('out.txt', 'w') as fh:
    for k in sorted(c.keys()):
        fh.write("{}={}\n".format(k, c[k]))

Data Files:

Tomato=78
Avocado=23
Pumpkin=100

Cucumber=17
Avocado=10
Cucumber=10

Solution: Write tests for script combining files

  • TBD
  • Because we have fixed paths in the script we have to create a directory structure that is similar to what is expected in a temporary location.
  • Run the script and compare the results to some expected file.
  • Then start refactoring the code.

Pytest resources

PyTest compare short lists - output

import configparser
import os


def test_read_ini(tmpdir):
    print(tmpdir)      # /private/var/folders/ry/z60xxmw0000gn/T/pytest-of-gabor/pytest-14/test_read0
    d = tmpdir.mkdir("subdir")
    fh = d.join("config.ini")
    fh.write("""
[application]
user  =  foo
password = secret
""")

    print(fh.basename) # data.txt
    print(fh.dirname)  # /private/var/folders/ry/z60xxmw0000gn/T/pytest-of-gabor/pytest-14/test_read0/subdir
    filename = os.path.join( fh.dirname, fh.basename )

    config = configparser.ConfigParser()
    config.read(filename)

    assert config.sections() == ['application']
    assert config['application'], {
       "user" : "foo",
       "password" : "secret"
    }

Anagram on the command line

from anagram import is_anagram
import sys

if len(sys.argv) != 3:
    exit("Usage {} STR STR".format(sys.argv[0]))

print(is_anagram(sys.argv[1], sys.argv[2]))

PyTest testing CLI

import subprocess

def capture(command):
    proc = subprocess.Popen(command,
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
    )
    out,err = proc.communicate()
    return out, err, proc.returncode


def test_anagram_no_param():
    command = ["python3", "examples/pytest/anagram.py"]
    out, err, exitcode = capture(command)
    assert exitcode == 1
    assert out == b''
    assert err == b'Usage examples/pytest/anagram.py STR STR\n'

def test_anagram():
    command = ["python3", "examples/pytest/anagram.py", "abc", "cba"]
    out, err, exitcode = capture(command)
    assert exitcode == 0
    assert out == b'True\n'
    assert err == b''

def test_no_anagram():
    command = ["python3", "examples/pytest/anagram.py", "abc", "def"]
    out, err, exitcode = capture(command)
    assert exitcode == 0
    assert out == b'False\n'
    assert err == b''

How to use the module?

Before we try to test this function let’s see how could we use it?

There is nothing special here, I just wanted to show it, because the testing is basically the same.


def add(x, y):
    """Adding two numbers

    >>> add(2, 2)
    4
    """

    return x * y

import mymath

print(mymath.add(2, 2))

from mymath import add

print(add(2, 2))

Pytest - simple passing test

We don’t need much to test such code. Just the following things:

  • Filename starts with test_
  • A function that starts with test_
  • Call the test function with some parameters and check if the results are as expected.

Specifically the assert function of Python expects to received a True (or False) value. If it is True the code keeps running as if nothing has happened.

If it is False and exception is raised.

import mymath

def test_add():
    assert mymath.add(2, 2) == 4

We can run the tests in two different ways. The regular would be to type in pytest and the name of the test file. In some setups this might not work and then we can also run python -m pytest and the name of the test file.

pytest test_mymath.py
python -m pytest test_mymath.py

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/math
plugins: flake8-1.0.6, dash-1.17.0
collected 1 item

test_mymath.py .                                                         [100%]

============================== 1 passed in 0.00s ===============================

The top of the output shows some information about the environment, (version numbers, plugins) then “collected” tells us how many test-cases were found by pytest. Each test function is one test case.

Then we see the name of the test file and a single dot indicating that there was one test-case and it was successful.

After the test run we could also see the exit code of the program by typing in echo $? on Linux or Mac or echo %ERRORLEVEL% on Windows.

$ echo $?
0

> echo %ERRORLEVEL%
0

Pytest failing test in one function

Once we had that passing test we might have shared our code just to receive complaints that it does not always work properly. One use might complain that passing in 2 and 3 does not give the expected 5.

So for your investigation the first thing you need to do is to write a test case expecting it to work proving that your code works. So you add a second assertion.


import mymath

def test_add():
    assert mymath.add(2, 2) == 4
    assert mymath.add(2, 3) == 5

To your surprise the tests fails with the following output:

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/math
plugins: flake8-1.0.6, dash-1.17.0
collected 1 item

test_mymath_more.py F                                                    [100%]

=================================== FAILURES ===================================
___________________________________ test_add ___________________________________

    def test_add():
        assert mymath.add(2, 2) == 4
>       assert mymath.add(2, 3) == 5
E       assert 6 == 5
E        +  where 6 = <function add at 0x7f6bc3c63160>(2, 3)
E        +    where <function add at 0x7f6bc3c63160> = mymath.add

test_mymath_more.py:6: AssertionError
=========================== short test summary info ============================
FAILED test_mymath_more.py::test_add - assert 6 == 5
============================== 1 failed in 0.02s ===============================

We see the collected 1 item because we still only have one test function.

Then next to the test file we see the letter F indicating that we had a single test failure.

Then we can see the details of the test failure. Among other things we can see the actual value returned by the add function and the expected value.

Knowing that assert only receives the True or False values of the comparison, you might wonder how did this happen. This is part of the magic of pytest. It uses some introspection to see what was in the expression that was passed to assert and it can print out the details helping us see what was the expected value and what was the actual value. This can help understanding the real problem behind the scenes.

You can also check the exit code and it will be something different from 0 indicating that something did not work. The exit code is used by CI-systems to see which test run were successful and which failed.

$ echo $?
1

> echo %ERRORLEVEL%
1

One big disadvantage of having two asserts in the same test function is that we don’t have clear indication that the first assert was successful. Moreover if the first assert fails then the second would not be even executed so we would not know what is the status of that case.

Pytest failing test separated

Instead of putting the two asserts in the same test function we could also put them in separate ones like in this example.


import mymath

def test_add():
    assert mymath.add(2, 2) == 4

def test_again():
    assert mymath.add(2, 3) == 5

The result of running this test file shows that it collected 2 items as there were two test functions.

Then next to the test file we see a dot indicating the successful test case and an F indicating the failed test. The more detailed test report helps.

At the bottom of the report you can also see that now it indicates 1 failed and 1 passed test.

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/gabor/work/slides/python/examples/pytest/math
plugins: flake8-1.0.6, dash-1.17.0
collected 2 items

test_mymath_more_separate.py .F                                          [100%]

=================================== FAILURES ===================================
__________________________________ test_again __________________________________

    def test_again():
>       assert mymath.add(2, 3) == 5
E       assert 6 == 5
E        +  where 6 = <function add at 0x7f4bfffa2c10>(2, 3)
E        +    where <function add at 0x7f4bfffa2c10> = mymath.add

test_mymath_more_separate.py:8: AssertionError
=========================== short test summary info ============================
FAILED test_mymath_more_separate.py::test_again - assert 6 == 5
========================= 1 failed, 1 passed in 0.03s ==========================