We learn a bunch of programming languages in school. Alongside these, we are told about debuggers and test libraries, but for such small assignments, it's much simpler to use print statements to debug the code. In fact, I've rarely had to use testing libraries or sanity checks in code I've written for an assignment (scratch that, I HAD to debug my CSC458 router implementation... curse those TCP/UDP packets -- GDB FTW!).
As projects mature, it is good to supplement the project's stable code with a test framework -- there are several good reasons to test and it's never too late to start. There are obvious pros to testing often but a team needs to be motivated to use it. For this, a test framework needs to have a few important qualities.
Note: These points are my opinion only and many people have their own views on the matter!
A test framework should:
- Have proper documentation
- Contain a simple "Start Tests" method or script
- Allow a simple and lightweight structure for developers to add their test cases
- Seamlessly integrate into existing code
- Allow features to be tested using real use cases
- Be as portable as the application it is testing
- Clearly output results for tests
- etc...
- Freeseer is written in Python, uses QT and GStreamer
- Freeseer's source tree is already organized into package format
- In the best case scenario, a test framework would impose little to no extra dependencies on Freeseer
Unittest vs. Py.Test
I didn't have to look very hard to find exhaustive comparisons of these two. A quick google search reveals several blog entries and documents from developers, Python enthusiasts and testers. I read a few, but the explanation that stood out to me was a series of blog entries found here and here.
The first link is about unittest, the second about Py.Test. For both, the author goes into great detail while focusing on: availability, ease of use, API complexity, test execution customization, test fixture management, test reuse and organization, assertion syntax, dealing with exceptions. The entries also have code snippets to show how everything is setups. At the end of each entry, the author wraps up with pros and cons for the framework just discussed.
If you're interested in general, I'd absolutely suggest reading the series, but to keep this short, I'll summarize my findings using relevant points to Freeseer.
Unittest
Unittest is part of the Python standard library and has been for quite some time. To use it, one needs to create a class which is of a unittest.TestCase. This module also facilitates a setup and teardown mechanism for test cases along with a suite() method to run several automated tests in sequence. To create a test suite, all tests must be imported and aggregated into a single main module. A test will pass or fail based on the result of assertions or explicit fails and tests result output is customizable. Here are some pros/cons from the author for unittest:
Pros
- available in the Python standard library
- easy to use by people familiar with the xUnit frameworks
- flexibility in test execution via command-line arguments
- support for test fixture/state management via set-up/tear-down hooks
- strong support for test organization and reuse via test suites
- xUnit flavor may be too strong for "pure" Pythonistas
- API can get in the way and can make the test code intent hard to understand
- tests can end up having a different look-and-feel from the code under test
- tests are executed in alphanumerical order
- assertions use custom syntax
Py.Test
"As Python unit test frameworks go, py.test is the new kid on the block. It already has an impressive set of features, with more to come, since the tool is under very active development."
Py.Test has no explicit API and automatically finds and runs tests (using prefix rules). Like unittest, there are setup and teardown mechanisms. Py.test has a huge collection of configurable command line parameters which customize the test execution. Since py.test automatically finds test cases, it makes it trivial to add a new test case to the suite (py.test can be configured to run all tests it finds in a given directory). Here are some pros and cons from the author:
Pros
- no API!
- great flexibility in test execution via command-line arguments
- strong support for test fixture/state management via setup/teardown hooks
- strong support for test organization via collection mechanism
- strong debugging support via customized traceback and assertion output
- very active and responsive development team
- many details, especially the ones related to customizing the collection process, are subject to refactorings and thus may change in the future
- a lot of magic goes on behind the scenes, which can sometimes obscure the tool's intent (it sure obscures its output sometimes)
For Freeseer, another con is that py.test is a third party dependency, but installation is as easy as:
$ easy_install pytest
Final Thoughts:
The first thing to note is that the blog posts are from 2005. There are things that py.test did then that unittest does now. For example, as of python 2.7, unittest now supports test discovery! Here's what that means:
$ python -m unittest discover /path/to/test_dir
will recursively go through /path/to/test_dir and will attempt to import all test_*.py modules. If successful and if the module contains a class which inherits from unittest.TestCase, all methods for this class will be executed. At the time of the blogs above, this was a beautiful feature of py.test which set it apart from unittest.
In python 2.7, unittest offers discovery, modular and logical grouping of test cases, informative verbose output (class name, method name, result) and a fail switch for the suite when a single unit test fails. In addition, we can run the test suite in discovery mode from the command line (like we did above) and as tests are executed information is outputted to the screen. Upon completion, the number of tests, time taken and result (OK/FAIL) are also outputted. All of this functionality is included into unittest which is in the python standard library and requires no external dependencies. Remember: LESS IS MORE. In all honesty, the only real problem with unittest for a project in general, is that it uses its own methods for assertions which pass/fail tests (it's not too complicated, just seems unnecessary).
To conclude, I am not saying unittest is better than py.test (because it isn't), but not all projects need py.test. The problem that py.test is trying to solve is compatibility in any code environment -- it does discovery and allows any method starting with test_* to be used as a test case. Though this is useful and simple, a project like Freeseer could actually benefit more from a structured module approach like unittest. Again, py.test can do this as well (in fact, unittest is clearly playing catch up) but it does so at the price of adding dependencies -- something Freeseer would rather not do.
Looking at the current organization of the source code in Freeseer, we have framework, frontend and plugins as the main separators. It would be simpler for developers to be able to create a test class for each class in the source and bundle together all unit tests for this in the test counterpart. This way, when functionality changes or gets extended, there is no guessing where the test code should go making it dead simple for developers to implement their own test case (BOOM no reason to not be able to test your code).
In the next post, I'll detail the design and show the power of using unittest in Freeseer to create a similar test code structure which will allow an obvious link between source modules and their test module counterparts.
Stay Tuned!
No comments:
Post a Comment