Software Design by Example: Running Tests

<p>
  <a href="https://third-bit.com/sdxpy/">Software Design by Example</a>
</p>
<h1>Running Tests</h1>
<div class="bottom">
  <a href="../">chapter</a>
</div>

---

## The Problem

-   Not all software needs formal testing
    -   Check one-off data analysis script incrementally
-   But 98% of the code in [SQLite][sqlite] is there to test the other 2%
    -   For which I am grateful
-   Good tools make tests easier to write
    -   So that programmers have fewer excuses not to write them
-   This lesson build a unit testing framework like [pytest][pytest]
    -   Most frameworks in most other languages share its design

---

## Functions in Lists

-   We can put functions in lists

```py
def first():
    print("First")

def second():
    print("Second")

def third():
    print("Third")

everything = [first, second, third]
for func in everything:
    func()
```

```
First
Second
Third
```

---

## Signatures

-   We have to know how to call the functions
    -   They must have the same <a class="gl-ref" href="../../glossary/#gl:signature" title="The ordered list of parameters and return values that specifies how a function must be called and what it returns." markdown="1">signature</a>

```py
def zero():
    print("zero")

def one(value):
    print("one", value)

for func in [zero, one]:
    func()
```

```
zero
Traceback (most recent call last):
  File "/sdx/test/signature.py", line 8, in <module>
    func()
TypeError: one() missing 1 required positional argument: 'value'
```

---

## Checking

-   Use `type` to see if something is a function

```py
print(type(3))
```

```
<class 'int'>
```

```py
def example():
    pass

print(type(example))
```

```
<class 'function'>
```

---

## Checking

-   But built-in functions have a different type

```py
print(type(len))
```

```
<class 'builtin_function_or_method'>
```

-   So use `callable` to check if something can be called

```py
def example():
    pass

print(callable(example), callable(len))
```

```
True True
```

---

## Testing Terminology

-   Apply the function we want to test to a <a class="gl-ref" href="../../glossary/#gl:fixture" title="The thing on which a test is run, such as the parameters to the function being tested or the file being processed." markdown="1">fixture</a>
-   Compare the <a class="gl-ref" href="../../glossary/#gl:actual_result" title="The value generated by running code in a test. If this matches the expected result, the test passes; if the two are different, the test fails." markdown="1">actual result</a>
    to the <a class="gl-ref" href="../../glossary/#gl:expected_result" title="The value that a piece of software is supposed to produce when tested in a certain way, or the state in which it is supposed to leave the system." markdown="1">expected result</a>
-   Possible outcomes are:
    -   <a class="gl-ref" href="../../glossary/#gl:pass_test" title="A test passes if the actual result matches the expected result." markdown="1">pass</a>: the target function worked
    -   <a class="gl-ref" href="../../glossary/#gl:fail_test" title="A test fails if the actual result does not match the expected result." markdown="1">fail</a>: the target function didn't do what we expected
    -   <a class="gl-ref" href="../../glossary/#gl:error_test" title="Signalled when something goes wrong in a unit test itself rather than in the system being tested. In this case, we do not know anything about the correctness of the system." markdown="1">error</a>: something went wrong with the test itself
-   Typically use `assert` to check results
    -   If condition is `True`, does nothing
    -   Otherwise, raises an `AssertionError`
-   Failed assertions usually cause the program to halt
    -   But we can catch the exception ourselves if we want

---

## A Function and Some Tests

```py
def sign(value):
    if value < 0:
        return -1
    else:
        return 1
```

```py
def test_sign_negative():
    assert sign(-3) == -1

def test_sign_positive():
    assert sign(19) == 1

def test_sign_zero():
    assert sign(0) == 0

def test_sign_error():
    assert sgn(1) == 1
```

---

## What We Want

```py
TESTS = [
    test_sign_negative,
    test_sign_positive,
    test_sign_zero,
    test_sign_error
]

run_tests(TESTS)
```

```
pass 2
fail 1
error 1
```

-   But we have to remember to add each one to `TESTS`

---

## How Python Stores Variables

-   Python stores variables in (something very much like) a dictionary

```py
import pprint
pprint.pprint(globals())
```

```
{'__annotations__': {},
 '__builtins__': <module 'builtins' (built-in)>,
 '__cached__': None,
 '__doc__': None,
 '__file__': '/sdx/test/globals.py',
 '__loader__': <_frozen_importlib_external.SourceFileLoader object \
at 0x109d65290>,
 '__name__': '__main__',
 '__package__': None,
 '__spec__': None,
 'pprint': <module 'pprint' from \
'/sdx/conda/envs/sdxpy/lib/python3.11/pprint.py'>}
```

---

## Further Proof

```py
import pprint
my_variable = 123
pprint.pprint(globals())
```

```
{'__annotations__': {},
 '__builtins__': <module 'builtins' (built-in)>,
 '__cached__': None,
 '__doc__': None,
 '__file__': '/sdx/test/globals_plus.py',
 '__loader__': <_frozen_importlib_external.SourceFileLoader object \
at 0x108039290>,
 '__name__': '__main__',
 '__package__': None,
 '__spec__': None,
 'my_variable': 123,
 'pprint': <module 'pprint' from \
'/sdx/conda/envs/sdxpy/lib/python3.11/pprint.py'>}
```

-   The function `locals` gives local variables

---

## Introspection

-   We know how to loop over a dictionary's keys

```py
def find_tests(prefix):
    for (name, func) in globals().items():
        if name.startswith(prefix):
            print(name, func)

find_tests("test_")
```

```
test_sign_negative <function test_sign_negative at 0x105bcd440>
test_sign_positive <function test_sign_positive at 0x105bcd4e0>
test_sign_zero <function test_sign_zero at 0x105bcd580>
test_sign_error <function test_sign_error at 0x105bcd620>
```

-   When we print a function, Python shows its name and address

---

## A Better Test Runner

```py
def run_tests():
    results = {"pass": 0, "fail": 0, "error": 0}
    for (name, test) in globals().items():
        if not name.startswith("test_"):
            continue
        try:
            test()
            results["pass"] += 1
        except AssertionError:
            results["fail"] += 1
        except Exception:
            results["error"] += 1
    print(f"pass {results['pass']}")
    print(f"fail {results['fail']}")
    print(f"error {results['error']}")
```

-   Really should check that tests are callable

---

## Summary

[pytest]: https://docs.pytest.org/
[sqlite]: https://sqlite.org/