Running Tests · Software Design by Example

The Problem

Not all software needs formal testing
- Check one-off data analysis script incrementally
But 98% of the code in SQLite is there to test the other 2%
- For which I am grateful
Good tools make tests easier to write
- So that programmers have fewer excuses not to write them
This lesson build a unit testing framework like pytest
- Most frameworks in most other languages share its design

Functions in Lists

We can put functions in lists

def first():
    print("First")

def second():
    print("Second")

def third():
    print("Third")

everything = [first, second, third]
for func in everything:
    func()

First
Second
Third

Signatures

We have to know how to call the functions
- They must have the same signature

def zero():
    print("zero")

def one(value):
    print("one", value)

for func in [zero, one]:
    func()

zero
Traceback (most recent call last):
  File "/sdx/test/signature.py", line 8, in <module>
    func()
TypeError: one() missing 1 required positional argument: 'value'

Checking

Use type to see if something is a function

print(type(3))

<class 'int'>

def example():
    pass

print(type(example))

<class 'function'>

Checking

But built-in functions have a different type

print(type(len))

<class 'builtin_function_or_method'>

So use callable to check if something can be called

def example():
    pass

print(callable(example), callable(len))

True True

Testing Terminology

Apply the function we want to test to a fixture
Compare the actual result to the expected result
Possible outcomes are:
- pass: the target function worked
- fail: the target function didn't do what we expected
- error: something went wrong with the test itself
Typically use assert to check results
- If condition is True, does nothing
- Otherwise, raises an AssertionError
Failed assertions usually cause the program to halt
- But we can catch the exception ourselves if we want

A Function and Some Tests

def sign(value):
    if value < 0:
        return -1
    else:
        return 1

def test_sign_negative():
    assert sign(-3) == -1

def test_sign_positive():
    assert sign(19) == 1

def test_sign_zero():
    assert sign(0) == 0

def test_sign_error():
    assert sgn(1) == 1

What We Want

TESTS = [
    test_sign_negative,
    test_sign_positive,
    test_sign_zero,
    test_sign_error
]

run_tests(TESTS)

pass 2
fail 1
error 1

But we have to remember to add each one to TESTS

How Python Stores Variables

Python stores variables in (something very much like) a dictionary

import pprint
pprint.pprint(globals())

{'__annotations__': {},
 '__builtins__': <module 'builtins' (built-in)>,
 '__cached__': None,
 '__doc__': None,
 '__file__': '/sdx/test/globals.py',
 '__loader__': <_frozen_importlib_external.SourceFileLoader object \
at 0x109d65290>,
 '__name__': '__main__',
 '__package__': None,
 '__spec__': None,
 'pprint': <module 'pprint' from \
'/sdx/conda/envs/sdxpy/lib/python3.11/pprint.py'>}

Further Proof

import pprint
my_variable = 123
pprint.pprint(globals())

{'__annotations__': {},
 '__builtins__': <module 'builtins' (built-in)>,
 '__cached__': None,
 '__doc__': None,
 '__file__': '/sdx/test/globals_plus.py',
 '__loader__': <_frozen_importlib_external.SourceFileLoader object \
at 0x108039290>,
 '__name__': '__main__',
 '__package__': None,
 '__spec__': None,
 'my_variable': 123,
 'pprint': <module 'pprint' from \
'/sdx/conda/envs/sdxpy/lib/python3.11/pprint.py'>}

The function locals gives local variables

Introspection

We know how to loop over a dictionary's keys

def find_tests(prefix):
    for (name, func) in globals().items():
        if name.startswith(prefix):
            print(name, func)

find_tests("test_")

test_sign_negative <function test_sign_negative at 0x105bcd440>
test_sign_positive <function test_sign_positive at 0x105bcd4e0>
test_sign_zero <function test_sign_zero at 0x105bcd580>
test_sign_error <function test_sign_error at 0x105bcd620>

When we print a function, Python shows its name and address

A Better Test Runner

def run_tests():
    results = {"pass": 0, "fail": 0, "error": 0}
    for (name, test) in globals().items():
        if not name.startswith("test_"):
            continue
        try:
            test()
            results["pass"] += 1
        except AssertionError:
            results["fail"] += 1
        except Exception:
            results["error"] += 1
    print(f"pass {results['pass']}")
    print(f"fail {results['fail']}")
    print(f"error {results['error']}")

Really should check that tests are callable

Summary

Concept map of unit testing framework — Figure 1: Concept map.