Topic: Regression Testing

Any time you modify an implementation within a program, you should also do regression testing. You can do so by rerunning existing tests against the modified code to determine whether the changes break anything that worked prior to the change and by writing new tests where necessary. Adequate coverage without wasting time should be a primary consideration when conducting regression tests. Try to spend as little time as possible doing regression testing without reducing the probability that you will detect new failures in old, already tested code.

Some strategies and factors to consider during this process include the following:

• Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.
• Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.
• Write a regression test for each bug fixed.
• If two or more tests are similar, determine which is less effective and get rid of it.
• Identify tests that the program consistently passes and archive them.
• Focus on functional issues, not those related to design.
• Make changes (small and large) to data and find any resulting corruption.
• Trace the effects of the changes on program memory.

Building a Library

The most effective approach to regression testing is based on developing a library of tests made up of a standard battery of test cases that can be run every time you build a new version of the program. The most difficult aspect involved in building a library of test cases is determining which test cases to include. The most common suggestion from authorities in the field of software testing is to avoid spending excessive amounts of time trying to decide and err on the side of caution. Automated tests, as well as test cases involving boundary conditions and timing almost definitely belong in your library. Some software development companies include only tests that have actually found bugs. The problem with that rationale is that the particular bug may have been found and fixed in the distant past.

Periodically review the regression test library to eliminate redundant or unnecessary tests. Do this about every third testing cycle. Duplication is quite common when more than one person is writing test code. An example that causes this problem is the concentration of tests that often develop when a bug or variants of it are particularly persistent and are present across many cycles of testing. Numerous tests might be written and added to the regression test library. These multiple tests are useful for fixing the bug, but when all traces of the bug and its variants are eliminated from the program, select the best of the tests associated with the bug and remove the rest from the library.