The built-in assertion methods are very useful, they are the bread and butter
of writing tests. However, soon enough you will probably want to write your
own assertions. Perhaps there are domain specific things that you want to
check (e.g. assert that two widgets are aligned parallel to the flux grid), or
perhaps you want to check something that could almost but not quite be found
in some other standard library (e.g. assert that two paths point to the same
file).
When you are in such situations, you could either make a base class for your
project that inherits from testtools.TestCase and make sure that all of
your tests derive from that, or you could use the testtools Matcher
system.
Using Matchers
Here’s a really basic example using stock matchers found in testtools:
import testtools
from testtools.matchers import Equals
class TestSquare(TestCase):
def test_square(self):
result = square(7)
self.assertThat(result, Equals(49))
The line self.assertThat(result, Equals(49)) is equivalent to
self.assertEqual(result, 49) and means “assert that result equals 49”.
The difference is that assertThat is a more general method that takes some
kind of observed value (in this case, result) and any matcher object
(here, Equals(49)).
The matcher object could be absolutely anything that implements the Matcher
protocol. This means that you can make more complex matchers by combining
existing ones:
def test_square_silly(self):
result = square(7)
self.assertThat(result, Not(Equals(50)))
Which is roughly equivalent to:
def test_square_silly(self):
result = square(7)
self.assertNotEqual(result, 50)
assert_that Function
In addition to self.assertThat, testtools also provides the assert_that
function in testtools.assertions This behaves like the method version does:
class TestSquare(TestCase):
def test_square():
result = square(7)
assert_that(result, Equals(49))
def test_square_silly():
result = square(7)
assert_that(result, Not(Equals(50)))
Delayed Assertions
A failure in the self.assertThat method will immediately fail the test: No
more test code will be run after the assertion failure.
The expectThat method behaves the same as assertThat with one
exception: when failing the test it does so at the end of the test code rather
than when the mismatch is detected. For example:
import subprocess
from testtools import TestCase
from testtools.matchers import Equals
class SomeProcessTests(TestCase):
def test_process_output(self):
process = subprocess.Popen(
["my-app", "/some/path"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
stdout, stderrr = process.communicate()
self.expectThat(process.returncode, Equals(0))
self.expectThat(stdout, Equals("Expected Output"))
self.expectThat(stderr, Equals(""))
In this example, should the expectThat call fail, the failure will be
recorded in the test result, but the test will continue as normal. If all
three assertions fail, the test result will have three failures recorded, and
the failure details for each failed assertion will be attached to the test
result.
Stock matchers
testtools comes with many matchers built in. They can all be found in and
imported from the testtools.matchers module.
Equals
Matches if two items are equal. For example:
def test_equals_example(self):
self.assertThat([42], Equals([42]))
Is
Matches if two items are identical. For example:
def test_is_example(self):
foo = object()
self.assertThat(foo, Is(foo))
IsInstance
Adapts isinstance() to use as a matcher. For example:
def test_isinstance_example(self):
class MyClass:pass
self.assertThat(MyClass(), IsInstance(MyClass))
self.assertThat(MyClass(), IsInstance(MyClass, str))
The raises helper
Matches if a callable raises a particular type of exception. For example:
def test_raises_example(self):
self.assertThat(lambda: 1/0, raises(ZeroDivisionError))
This is actually a convenience function that combines two other matchers:
Raises and MatchesException.
DocTestMatches
Matches a string as if it were the output of a doctest example. Very useful
for making assertions about large chunks of text. For example:
import doctest
def test_doctest_example(self):
output = "Colorless green ideas"
self.assertThat(
output,
DocTestMatches("Colorless ... ideas", doctest.ELLIPSIS))
We highly recommend using the following flags:
doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_NDIFF
GreaterThan
Matches if the given thing is greater than the thing in the matcher. For
example:
def test_greater_than_example(self):
self.assertThat(3, GreaterThan(2))
LessThan
Matches if the given thing is less than the thing in the matcher. For
example:
def test_less_than_example(self):
self.assertThat(2, LessThan(3))
StartsWith, EndsWith
These matchers check to see if a string starts with or ends with a particular
substring. For example:
def test_starts_and_ends_with_example(self):
self.assertThat('underground', StartsWith('und'))
self.assertThat('underground', EndsWith('und'))
Contains
This matcher checks to see if the given thing contains the thing in the
matcher. For example:
def test_contains_example(self):
self.assertThat('abc', Contains('b'))
MatchesException
Matches an exc_info tuple if the exception is of the correct type. For
example:
def test_matches_exception_example(self):
try:
raise RuntimeError('foo')
except RuntimeError:
exc_info = sys.exc_info()
self.assertThat(exc_info, MatchesException(RuntimeError))
self.assertThat(exc_info, MatchesException(RuntimeError('bar')))
Most of the time, you will want to uses The raises helper instead.
NotEquals
Matches if something is not equal to something else. Note that this is subtly
different to Not(Equals(x)). NotEquals(x) will match if y != x,
Not(Equals(x)) will match if not y == x.
You only need to worry about this distinction if you are testing code that
relies on badly written overloaded equality operators.
KeysEqual
Matches if the keys of one dict are equal to the keys of another dict. For
example:
def test_keys_equal(self):
x = {'a': 1, 'b': 2}
y = {'a': 2, 'b': 3}
self.assertThat(x, KeysEqual(y))
MatchesRegex
Matches a string against a regular expression, which is a wonderful thing to
be able to do, if you think about it:
def test_matches_regex_example(self):
self.assertThat('foo', MatchesRegex('fo+'))
HasLength
Check the length of a collection. The following assertion will fail:
self.assertThat([1, 2, 3], HasLength(2))
But this one won’t:
self.assertThat([1, 2, 3], HasLength(3))
Combining matchers
One great thing about matchers is that you can readily combine existing
matchers to get variations on their behaviour or to quickly build more complex
assertions.
Below are a few of the combining matchers that come with testtools.
Not
Negates another matcher. For example:
def test_not_example(self):
self.assertThat([42], Not(Equals("potato")))
self.assertThat([42], Not(Is([42])))
If you find yourself using Not frequently, you may wish to create a custom
matcher for it. For example:
IsNot = lambda x: Not(Is(x))
def test_not_example_2(self):
self.assertThat([42], IsNot([42]))
Annotate
Used to add custom notes to a matcher. For example:
def test_annotate_example(self):
result = 43
self.assertThat(
result, Annotate("Not the answer to the Question!", Equals(42)))
Since the annotation is only ever displayed when there is a mismatch
(e.g. when result does not equal 42), it’s a good idea to phrase the note
negatively, so that it describes what a mismatch actually means.
As with Not, you may wish to create a custom matcher that describes a
common operation. For example:
PoliticallyEquals = lambda x: Annotate("Death to the aristos!", Equals(x))
def test_annotate_example_2(self):
self.assertThat("orange", PoliticallyEquals("yellow"))
You can have assertThat perform the annotation for you as a convenience:
def test_annotate_example_3(self):
self.assertThat("orange", Equals("yellow"), "Death to the aristos!")
AfterPreprocessing
Used to make a matcher that applies a function to the matched object before
matching. This can be used to aid in creating trivial matchers as functions, for
example:
def test_after_preprocessing_example(self):
def PathHasFileContent(content):
def _read(path):
return open(path).read()
return AfterPreprocessing(_read, Equals(content))
self.assertThat('/tmp/foo.txt', PathHasFileContent("Hello world!"))
MatchesAll
Combines many matchers to make a new matcher. The new matcher will only match
things that match every single one of the component matchers.
It’s much easier to understand in Python than in English:
def test_matches_all_example(self):
has_und_at_both_ends = MatchesAll(StartsWith("und"), EndsWith("und"))
# This will succeed.
self.assertThat("underground", has_und_at_both_ends)
# This will fail.
self.assertThat("found", has_und_at_both_ends)
# So will this.
self.assertThat("undead", has_und_at_both_ends)
At this point some people ask themselves, “why bother doing this at all? why
not just have two separate assertions?”. It’s a good question.
The first reason is that when a MatchesAll gets a mismatch, the error will
include information about all of the bits that mismatched. When you have two
separate assertions, as below:
def test_two_separate_assertions(self):
self.assertThat("foo", StartsWith("und"))
self.assertThat("foo", EndsWith("und"))
Then you get absolutely no information from the second assertion if the first
assertion fails. Tests are largely there to help you debug code, so having
more information in error messages is a big help.
The second reason is that it is sometimes useful to give a name to a set of
matchers. has_und_at_both_ends is a bit contrived, of course, but it is
clear. The FileExists and DirExists matchers included in testtools
are perhaps better real examples.
If you want only the first mismatch to be reported, pass first_only=True
as a keyword parameter to MatchesAll.
MatchesAny
Like MatchesAll, MatchesAny combines many matchers to make a new
matcher. The difference is that the new matchers will match a thing if it
matches any of the component matchers.
For example:
def test_matches_any_example(self):
self.assertThat(42, MatchesAny(Equals(5), Not(Equals(6))))
AllMatch
Matches many values against a single matcher. Can be used to make sure that
many things all meet the same condition:
def test_all_match_example(self):
self.assertThat([2, 3, 5, 7], AllMatch(LessThan(10)))
If the match fails, then all of the values that fail to match will be included
in the error message.
In some ways, this is the converse of MatchesAll.
MatchesListwise
Where MatchesAny and MatchesAll combine many matchers to match a
single value, MatchesListwise combines many matches to match many values.
For example:
def test_matches_listwise_example(self):
self.assertThat(
[1, 2, 3], MatchesListwise(map(Equals, [1, 2, 3])))
This is useful for writing custom, domain-specific matchers.
If you want only the first mismatch to be reported, pass first_only=True
to MatchesListwise.
MatchesSetwise
Combines many matchers to match many values, without regard to their order.
Here’s an example:
def test_matches_setwise_example(self):
self.assertThat(
[1, 2, 3], MatchesSetwise(Equals(2), Equals(3), Equals(1)))
Much like MatchesListwise, best used for writing custom, domain-specific
matchers.
MatchesStructure
Creates a matcher that matches certain attributes of an object against a
pre-defined set of matchers.
It’s much easier to understand in Python than in English:
def test_matches_structure_example(self):
foo = Foo()
foo.a = 1
foo.b = 2
matcher = MatchesStructure(a=Equals(1), b=Equals(2))
self.assertThat(foo, matcher)
Since all of the matchers used were Equals, we could also write this using
the byEquality helper:
def test_matches_structure_example(self):
foo = Foo()
foo.a = 1
foo.b = 2
matcher = MatchesStructure.byEquality(a=1, b=2)
self.assertThat(foo, matcher)
MatchesStructure.fromExample takes an object and a list of attributes and
creates a MatchesStructure matcher where each attribute of the matched
object must equal each attribute of the example object. For example:
matcher = MatchesStructure.fromExample(foo, 'a', 'b')
is exactly equivalent to matcher in the previous example.
MatchesPredicate
Sometimes, all you want to do is create a matcher that matches if a given
function returns True, and mismatches if it returns False.
For example, you might have an is_prime function and want to make a
matcher based on it:
def test_prime_numbers(self):
IsPrime = MatchesPredicate(is_prime, '%s is not prime.')
self.assertThat(7, IsPrime)
self.assertThat(1983, IsPrime)
# This will fail.
self.assertThat(42, IsPrime)
Which will produce the error message:
Traceback (most recent call last):
File "...", line ..., in test_prime_numbers
self.assertThat(42, IsPrime)
MismatchError: 42 is not prime.
MatchesPredicateWithParams
Sometimes you can’t use a trivial predicate and instead need to pass in some
parameters each time. In that case, MatchesPredicateWithParams is your go-to
tool for creating ad hoc matchers. MatchesPredicateWithParams takes a predicate
function and message and returns a factory to produce matchers from that. The
predicate needs to return a boolean (or any truthy object), and accept the
object to match + whatever was passed into the factory.
For example, you might have an divisible function and want to make a
matcher based on it:
def test_divisible_numbers(self):
IsDivisibleBy = MatchesPredicateWithParams(
divisible, '{0} is not divisible by {1}')
self.assertThat(7, IsDivisibleBy(1))
self.assertThat(7, IsDivisibleBy(7))
self.assertThat(7, IsDivisibleBy(2))
# This will fail.
Which will produce the error message:
Traceback (most recent call last):
File "...", line ..., in test_divisible
self.assertThat(7, IsDivisibleBy(2))
MismatchError: 7 is not divisible by 2.
Raises
Takes whatever the callable raises as an exc_info tuple and matches it against
whatever matcher it was given. For example, if you want to assert that a
callable raises an exception of a given type:
def test_raises_example(self):
self.assertThat(
lambda: 1/0, Raises(MatchesException(ZeroDivisionError)))
Although note that this could also be written as:
def test_raises_example_convenient(self):
self.assertThat(lambda: 1/0, raises(ZeroDivisionError))
See also MatchesException and the raises helper
Writing your own matchers
Combining matchers is fun and can get you a very long way indeed, but
sometimes you will have to write your own. Here’s how.
You need to make two closely-linked objects: a Matcher and a
Mismatch. The Matcher knows how to actually make the comparison, and
the Mismatch knows how to describe a failure to match.
Here’s an example matcher:
class IsDivisibleBy(object):
"""Match if a number is divisible by another number."""
def __init__(self, divider):
self.divider = divider
def __str__(self):
return 'IsDivisibleBy(%s)' % (self.divider,)
def match(self, actual):
remainder = actual % self.divider
if remainder != 0:
return IsDivisibleByMismatch(actual, self.divider, remainder)
else:
return None
The matcher has a constructor that takes parameters that describe what you
actually expect, in this case a number that other numbers ought to be
divisible by. It has a __str__ method, the result of which is displayed
on failure by assertThat and a match method that does the actual
matching.
match takes something to match against, here actual, and decides
whether or not it matches. If it does match, then match must return
None. If it does not match, then match must return a Mismatch
object. assertThat will call match and then fail the test if it
returns a non-None value. For example:
def test_is_divisible_by_example(self):
# This succeeds, since IsDivisibleBy(5).match(10) returns None.
self.assertThat(10, IsDivisibleBy(5))
# This fails, since IsDivisibleBy(7).match(10) returns a mismatch.
self.assertThat(10, IsDivisibleBy(7))
The mismatch is responsible for what sort of error message the failing test
generates. Here’s an example mismatch:
class IsDivisibleByMismatch(object):
def __init__(self, number, divider, remainder):
self.number = number
self.divider = divider
self.remainder = remainder
def describe(self):
return "%r is not divisible by %r, %r remains" % (
self.number, self.divider, self.remainder)
def get_details(self):
return {}
The mismatch takes information about the mismatch, and provides a describe
method that assembles all of that into a nice error message for end users.
You can use the get_details method to provide extra, arbitrary data with
the mismatch (e.g. the contents of a log file). Most of the time it’s fine to
just return an empty dict. You can read more about Details elsewhere in this
document.
Sometimes you don’t need to create a custom mismatch class. In particular, if
you don’t care when the description is calculated, then you can just do that
in the Matcher itself like this:
def match(self, actual):
remainder = actual % self.divider
if remainder != 0:
return Mismatch(
"%r is not divisible by %r, %r remains" % (
actual, self.divider, remainder))
else:
return None
When writing a describe method or constructing a Mismatch object the
code should ensure it only emits printable unicode. As this output must be
combined with other text and forwarded for presentation, letting through
non-ascii bytes of ambiguous encoding or control characters could throw an
exception or mangle the display. In most cases simply avoiding the %s
format specifier and using %r instead will be enough. For examples of
more complex formatting see the testtools.matchers implementatons.