Topic: Evaluate Test Cases

A suite of system test cases can find many defects, but still leave many other critical defects undetected. One clear way to guard against undetected defects is to increase the coverage of your test suite.

While a suite of unit tests might be evaluated in terms of its implementation coverage, a suite of system test cases should instead be evaluated in terms of specification coverage. Implementation coverage measures the percentage of lines of code that are executed by the unit test cases. If there is a line of code that is never executed, then there could be an undetected defect on that line. Specification coverage measures the percentage of written requirements that the system test suite covers. If there is a requirement that is not tested by any system test case, then you are not assured that the requirement has been satisfied.

You can evaluate the coverage of your system tests on two levels. First, the test suite itself is an organized table of contents for the test cases that can make it easy to notice parts of the system that are not being tested. Second, within an individual test case, the set of possible input values should cover all input value equivalence partitions for each parameter.