Python library 'unittest': Generate multiple tests programmatically [duplicate]

This question already has answers here: Closed 10 years ago.

Possible Duplicate:

How do you generate dynamic (parameterized) unit tests in Python?

I have a function to test, under_test, and a set of expected input/output pairs:

[
(2, 332),
(234, 99213),
(9, 3),
# ...
]

I would like each one of these input/output pairs to be tested in its own test_* method. Is that possible?

This is sort of what I want, but forcing every single input/output pair into a single test:

class TestPreReqs(unittest.TestCase):

    def setUp(self):
        self.expected_pairs = [(23, 55), (4, 32)]

    def test_expected(self):
        for exp in self.expected_pairs:
            self.assertEqual(under_test(exp[0]), exp[1])

if __name__ == '__main__':
    unittest.main()

(Also, do I really want to be putting that definition of self.expected_pairs in setUp?)

UPDATE: Trying doublep's advice:

class TestPreReqs(unittest.TestCase):

    def setUp(self):
        expected_pairs = [
                          (2, 3),
                          (42, 11),
                          (3, None),
                          开发者_Python百科(31, 99),
                         ]

        for k, pair in expected_pairs:
            setattr(TestPreReqs, 'test_expected_%d' % k, create_test(pair))

    def create_test (pair):
        def do_test_expected(self):
            self.assertEqual(get_pre_reqs(pair[0]), pair[1])
        return do_test_expected


if __name__ == '__main__':
    unittest.main()

This does not work. 0 tests are run. Did I adapt the example incorrectly?

---

I had to do something similar. I created simple TestCase subclasses that took a value in their __init__, like this:

class KnownGood(unittest.TestCase):
    def __init__(self, input, output):
        super(KnownGood, self).__init__()
        self.input = input
        self.output = output
    def runTest(self):
        self.assertEqual(function_to_test(self.input), self.output)

I then made a test suite with these values:

def suite():
    suite = unittest.TestSuite()
    suite.addTests(KnownGood(input, output) for input, output in known_values)
    return suite

You can then run it from your main method:

if __name__ == '__main__':
    unittest.TextTestRunner().run(suite())

The advantages of this are:

• As you add more values, the number of reported tests increases, which makes you feel like you are doing more.
• Each individual test case can fail individually
• It's conceptually simple, since each input/output value is converted into one TestCase

---

Not tested:

class TestPreReqs(unittest.TestCase):
    ...

def create_test (pair):
    def do_test_expected(self):
        self.assertEqual(under_test(pair[0]), pair[1])
    return do_test_expected

for k, pair in enumerate ([(23, 55), (4, 32)]):
    test_method = create_test (pair)
    test_method.__name__ = 'test_expected_%d' % k
    setattr (TestPreReqs, test_method.__name__, test_method)

If you use this often, you could prettify this by using utility functions and/or decorators, I guess. Note that pairs are not an attribute of TestPreReqs object in this example (and so setUp is gone). Rather, they are "hardwired" in a sense to the TestPreReqs class.

---

As often with Python, there is a complicated way to provide a simple solution.

In that case, we can use metaprogramming, decorators, and various nifty Python tricks to achieve a nice result. Here is what the final test will look like:

import unittest

# Some magic code will be added here later

class DummyTest(unittest.TestCase):
  @for_examples(1, 2)
  @for_examples(3, 4)
  def test_is_smaller_than_four(self, value):
    self.assertTrue(value < 4)

  @for_examples((1,2),(2,4),(3,7))
  def test_double_of_X_is_Y(self, x, y):
    self.assertEqual(2 * x, y)

if __name__ == "__main__":
  unittest.main()

When executing this script, the result is:

..F...F
======================================================================
FAIL: test_double_of_X_is_Y(3,7)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 41, in test_double_of_X_is_Y
    self.assertEqual(2 * x, y)
AssertionError: 6 != 7

======================================================================
FAIL: test_is_smaller_than_four(4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 37, in test_is_smaller_than_four
    self.assertTrue(value < 4)
AssertionError

----------------------------------------------------------------------
Ran 7 tests in 0.001s

FAILED (failures=2)

which achieves our goal:

• it is unobtrusive: we derive from TestCase as usual
• we write parametrized tests only once
• each example value is considered an individual test
• the decorator can be stacked, so it is easy to use sets of examples (e.g., using a function to build the list of values from example files or directories)
• The icing on the cake is it works for arbitrary arity of the signature

So how does it work? Basically, the decorator stores the examples in an attribute of the function. We use a metaclass to replace every decorated function with a list of functions. And we replace the unittest.TestCase with our new magic code (to be pasted in the "magic" comment above) is:

__examples__ = "__examples__"

def for_examples(*examples):
    def decorator(f, examples=examples):
      setattr(f, __examples__, getattr(f, __examples__,()) + examples)
      return f
    return decorator

class TestCaseWithExamplesMetaclass(type):
  def __new__(meta, name, bases, dict):
    def tuplify(x):
      if not isinstance(x, tuple):
        return (x,)
      return x
    for methodname, method in dict.items():
      if hasattr(method, __examples__):
        dict.pop(methodname)
        examples = getattr(method, __examples__)
        delattr(method, __examples__)
        for example in (tuplify(x) for x in examples):
          def method_for_example(self, method = method, example = example):
            method(self, *example)
          methodname_for_example = methodname + "(" + ", ".join(str(v) for v in example) + ")"
          dict[methodname_for_example] = method_for_example
    return type.__new__(meta, name, bases, dict)

class TestCaseWithExamples(unittest.TestCase):
  __metaclass__ = TestCaseWithExamplesMetaclass
  pass

unittest.TestCase = TestCaseWithExamples

If someone wants to package this nicely, or propose a patch for unittest, feel free! A quote of my name will be appreciated.

The code can be made much simpler and fully encapsulated in the decorator if you are ready to use frame introspection (import the sys module)

def for_examples(*parameters):

  def tuplify(x):
    if not isinstance(x, tuple):
      return (x,)
    return x

  def decorator(method, parameters=parameters):
    for parameter in (tuplify(x) for x in parameters):

      def method_for_parameter(self, method=method, parameter=parameter):
        method(self, *parameter)
      args_for_parameter = ",".join(repr(v) for v in parameter)
      name_for_parameter = method.__name__ + "(" + args_for_parameter + ")"
      frame = sys._getframe(1)  # pylint: disable-msg=W0212
      frame.f_locals[name_for_parameter] = method_for_parameter
    return None
  return decorator

---

nose (suggested by @Paul Hankin)

#!/usr/bin/env python
# file: test_pairs_nose.py
from nose.tools import eq_ as eq

from mymodule import f

def test_pairs():
    for input, output in [ (2, 332), (234, 99213), (9, 3), ]:
        yield _test_f, input, output

def _test_f(input, output):
    try:
        eq(f(input), output)
    except AssertionError:
        if input == 9: # expected failure
            from nose.exc import SkipTest
            raise SkipTest("expected failure")
        else:
            raise

if __name__=="__main__":
   import nose; nose.main()

Example:

$ nosetests test_pairs_nose -v
test_pairs_nose.test_pairs(2, 332) ... ok
test_pairs_nose.test_pairs(234, 99213) ... ok
test_pairs_nose.test_pairs(9, 3) ... SKIP: expected failure

----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK (SKIP=1)

unittest (an approach similar to @doublep's one)

#!/usr/bin/env python
import unittest2 as unittest
from mymodule import f

def add_tests(generator):
    def class_decorator(cls):
        """Add tests to `cls` generated by `generator()`."""
        for f, input, output in generator():
            test = lambda self, i=input, o=output, f=f: f(self, i, o)
            test.__name__ = "test_%s(%r, %r)" % (f.__name__, input, output)
            setattr(cls, test.__name__, test)
        return cls
    return class_decorator

def _test_pairs():
    def t(self, input, output):
        self.assertEqual(f(input), output)

    for input, output in [ (2, 332), (234, 99213), (9, 3), ]:
        tt = t if input != 9 else unittest.expectedFailure(t)
        yield tt, input, output

class TestCase(unittest.TestCase):
    pass
TestCase = add_tests(_test_pairs)(TestCase)

if __name__=="__main__":
    unittest.main()

Example:

$ python test_pairs_unit2.py -v
test_t(2, 332) (__main__.TestCase) ... ok
test_t(234, 99213) (__main__.TestCase) ... ok
test_t(9, 3) (__main__.TestCase) ... expected failure

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK (expected failures=1)

If you don't want to install unittest2 then add:

try:
    import unittest2 as unittest
except ImportError:
    import unittest
    if not hasattr(unittest, 'expectedFailure'):
       import functools
       def _expectedFailure(func):
           @functools.wraps(func)
           def wrapper(*args, **kwargs):
               try:
                   func(*args, **kwargs)
               except AssertionError:
                   pass
               else:
                   raise AssertionError("UnexpectedSuccess")
           return wrapper
       unittest.expectedFailure = _expectedFailure

---

Some of the tools available for doing parametrized tests in Python are:

• Nose test generators (only for function tests, not TestCase classes)
• nose-parametrized by David Wolever (also for TestCase classes)
• Unittest template by Boris Feld
• Parametrized tests in py.test
• parametrized-testcase by Austin Bingham

See also question 1676269 for more answers to this question.

---

I think Rory's solution is the cleanest and shortest. However, this variation of doublep's "create synthetic functions in a TestCase" also works:

from functools import partial

class TestAllReports(unittest.TestCase):
    pass

def test_spamreport(name):
    assert classify(getSample(name))=='spamreport', name

for rep in REPORTS:
    testname = 'test_' + rep
    testfunc = partial(test_spamreport, rep)
    testfunc.__doc__ = testname
    setattr(TestAllReports, testname, testfunc)

if __name__=='__main__':
    unittest.main(argv=sys.argv + ['--verbose'])