testrunner Edge Cases ===================== This document has some edge-case examples to test various aspects of the test runner. Separating Python path and test directories ------------------------------------------- The --path option defines a directory to be searched for tests *and* a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path: >>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> from zope.testing import testrunner >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Test-module import failures: Module: sampletestsf Traceback (most recent call last): ImportError: No module named sampletestsf ... >>> sys.path.append(directory_with_tests) >>> sys.argv = ['test'] >>> testrunner.run_internal(defaults) ... # doctest: +ELLIPSIS Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures and 0 errors in 0.000 seconds. ... Running zope.testing.testrunner.layer.UnitTests tests: Tear down samplelayers.Layer122 in N.NNN seconds. Tear down samplelayers.Layer12 in N.NNN seconds. Tear down samplelayers.Layer1 in N.NNN seconds. Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Ran 192 tests with 0 failures and 0 errors in N.NNN seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Total: 405 tests, 0 failures, 0 errors in N.NNN seconds. False Bug #251759: The test runner's protection against descending into non-package directories was ineffective, e.g. picking up tests from eggs that were stored close by: >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex-251759') >>> defaults = [ ... '--test-path', directory_with_tests, ... ] >>> testrunner.run_internal(defaults) Total: 0 tests, 0 failures, 0 errors in 0.000 seconds. False Debugging Edge Cases -------------------- >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex') >>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print line ... return line+'\n' >>> real_stdin = sys.stdin Using pdb.set_trace in a function called by an ordinary test: >>> if sys.version_info[:2] == (2, 3): ... sys.stdin = Input('n\np x\nc') ... else: ... sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace2').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running zope.testing.testrunner.layer.UnitTests tests:... > testrunner-ex/sample3/sampletests_d.py(47)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.001 seconds. ... False Using pdb.set_trace in a function called by a doctest in a doc string: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace4').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testing.testrunner.layer.UnitTests tests:... --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > testrunner-ex/sample3/sampletests_d.py(42)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. ... False Using pdb in a docstring-based doctest >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace3').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testing.testrunner.layer.UnitTests tests:... --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > (3)...() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. ... False Using pdb.set_trace in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace5').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testing.testrunner.layer.UnitTests tests:... --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > (3)...() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. ... False Using pdb.set_trace in a function called by a doctest in a doc file: >>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace6').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin Running zope.testing.testrunner.layer.UnitTests tests:... --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > testrunner-ex/sample3/sampletests_d.py(42)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. ... False Post-mortem debugging function called from ordinary test: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests:... Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 37, in test_post_mortem2 g() File "testrunner-ex/sample3/sampletests_d.py", line 46, in g raise ValueError ValueError exceptions.ValueError: > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True Post-mortem debugging docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem3 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem3 (sample3.sampletests_d) Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/sampletests_d.py:61 (2 examples)> exceptions.ValueError: > (1)...() (Pdb) p x 1 (Pdb) c True Post-mortem debugging function called from docstring-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem4 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Error in test post_mortem4 (sample3.sampletests_d) Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/sampletests_d.py:NNN (1 example)> exceptions.ValueError: > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True Post-mortem debugging file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem5 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem5.txt Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)> exceptions.ValueError: > (1)...() (Pdb) p x 1 (Pdb) c True Post-mortem debugging function called from file-based doctest: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem6 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests:... Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds. Error testrunner-ex/sample3/post_mortem6.txt Traceback (most recent call last): File ".../zope/testing/doctest/__init__.py", Line NNN, in debug runner.run(self._dt_test, clear_globs=False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run r = DocTestRunner.run(self, test, compileflags, out, False) File ".../zope/testing/doctest/__init__.py", Line NNN, in run return self.__run(test, compileflags, out) File ".../zope/testing/doctest/__init__.py", Line NNN, in __run exc_info) File ".../zope/testing/doctest/__init__.py", Line NNN, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)> exceptions.ValueError: > testrunner-ex/sample3/sampletests_d.py(NNN)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True Post-mortem debugging of a docstring doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure2 -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests:... Error in test post_mortem_failure2 (sample3.sampletests_d) File "testrunner-ex/sample3/sampletests_d.py", line 81, in sample3.sampletests_d.post_mortem_failure2 x Want: 2 Got: 1 > testrunner-ex/sample3/sampletests_d.py(81)_() exceptions.ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c True Post-mortem debugging of a docfile doctest failure: >>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure.txt -D').split() >>> try: testrunner.run_internal(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running zope.testing.testrunner.layer.UnitTests tests:... Error in test /home/jim/z3/zope.testing/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt File "testrunner-ex/sample3/post_mortem_failure.txt", line 2, in post_mortem_failure.txt x Want: 2 Got: 1 > testrunner-ex/sample3/post_mortem_failure.txt(2)_() exceptions.ValueError: Expected and actual output are different > (1)...() (Pdb) p x 1 (Pdb) c True Post-mortem debugging with triple verbosity >>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split() >>> testrunner.run_internal(defaults) Running tests at level 1 Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Running: test_x1 (sampletestsf.TestA1) (0.000 s) test_y0 (sampletestsf.TestA1) (0.000 s) test_z0 (sampletestsf.TestA1) (0.000 s) test_x0 (sampletestsf.TestB1) (0.000 s) test_y1 (sampletestsf.TestB1) (0.000 s) test_z0 (sampletestsf.TestB1) (0.000 s) test_1 (sampletestsf.TestNotMuch1) (0.000 s) test_2 (sampletestsf.TestNotMuch1) (0.000 s) test_3 (sampletestsf.TestNotMuch1) (0.000 s) Ran 9 tests with 0 failures and 0 errors in 0.001 seconds. Tearing down left over layers: Tear down samplelayers.Layer1 in 0.000 seconds. False Test Suites with None for suites or tests ----------------------------------------- >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_suite$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_suite Traceback (most recent call last): TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite Test-modules with import problems: sample1.sampletests_none_suite Total: 0 tests, 0 failures, 0 errors in N.NNN seconds. True >>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_test$', ... ] >>> testrunner.run_internal(defaults) Test-module import failures: Module: sample1.sampletests_none_test Traceback (most recent call last): TypeError: ... Test-modules with import problems: sample1.sampletests_none_test Total: 0 tests, 0 failures, 0 errors in N.NNN seconds. True You must use --repeat with --report-refcounts --------------------------------------------- It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1 >>> sys.argv = 'test -r'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True >>> sys.argv = 'test -r -N1'.split() >>> testrunner.run_internal(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. True