|
AUTOMATED TESTING - WHY AND HOW
Everyone knows how important testing is, and, with luck, everyone actually does test the software that they release. But do they really? Can they? Even a simple program often has many different possible behaviors, some of which only take place in rather unusual (and hard to duplicate) circumstances. Even if every possible behavior was tested when the program was first released to the users, what about the second release, or even a "minor" modification? The feature being modified will probably be re-tested, but what about other, seemingly unrelated, features that may have been inadvertently broken by the modification? Will every unusual test case from the first release's testing be remembered, much less retried, for the new release, especially if retrying the test would require a lot of preliminary work (e.g. adding appropriate test records to the database)?
This problem arose for us several years ago, when we found that our software was getting so complicated that testing everything before release was a real chore, and a good many bugs (some of them very obvious) were getting out into the field. What's more, I found that I was actually afraid to add new features, concerned that they might break the rest of the software. It was this last problem that really drove home to me the importance of making it possible to quickly and easily test all the features of all our products.
AUTOMATED TESTING
The principle of automated testing is that there is a program (which could be a job stream) that runs the program being tested, feeding it the proper input, and checking the output against the output that was expected. Once the test suite is written, no human intervention is needed, either to run the program or to look to see if it worked; the test suite does all that, and somehow indicates (say, by a :TELL message and a results file) whether the program's output was as expected. We, for instance, have over two hundred test suites, all of which can be run overnight by executing one job stream submission command; after they run, another command can show which test suites succeeded and which failed.
These test suites can help in many ways:
* As discussed above, the test suites should always be run before a new version is released, no matter how trivial the modifications to the program.
* If the software is internally different for different environments (e.g. MPE/V vs. MPE/XL), but should have the same external behavior, the test suites should be run on both environments.
* As you're making serious changes to the software, you might want to run the test suites even before the release, since they can tell you what still needs to be fixed.
* If you have the discipline to -- believe it or not -- write the test suite before you've written your program, you can even use the test suite to do the initial testing of your code. After all, you'd have to initially test the code anyway; you might as well use your test suites to do that initial testing as well as all subsequent tests.
Note also that the test suites not only run the program, but set up the proper environment for the program; this might mean filling up a test database, building necessary files, etc.
rgds
rose
|