Everyone  knows  how  important  testing is, and,  with luck, everyone
actually does test the software that they release. But do they really?
Can  they?  Even  a  simple program often  has many different possible
behaviors,  some of which only take  place in rather unusual (and hard
to  duplicate)  circumstances.  Even  if  every possible  behavior was
tested  when  the program was first released  to the users, what about
the  second release, or even a "minor" modification? The feature being
modified  will probably be re-tested,  but what about other, seemingly
unrelated,  features  that  may have been  inadvertently broken by the
modification?  Will  every unusual test case  from the first release's
testing  be  remembered,  much  less  retried,  for  the  new release,
especially  if  retrying  the test would require  a lot of preliminary
work (e.g. adding appropriate test records to the database)?

This  problem  arose for us several years  ago, when we found that our
software  was  getting  so complicated that  testing everything before
release  was  a  real  chore, and a good many  bugs (some of them very
obvious)  were getting out into the field. What's more, I found that I
was  actually  afraid  to add new features,  concerned that they might
break  the rest of the software. It  was this last problem that really
drove  home to me the importance of  making it possible to quickly and

The  principle of automated testing is  that there is a program (which
could  be a job stream) that runs the program being tested, feeding it
the  proper input, and checking the output against the output that was
expected.  Once  the  test suite is written,  no human intervention is
needed,  either to run the program or to look to see if it worked; the
test  suite  does  all  that,  and somehow indicates  (say, by a :TELL
message  and  a  results  file)  whether  the program's  output was as
expected.  We, for instance, have over two hundred test suites, all of
which  can  be  run  overnight by executing  one job stream submission
command;  after  they run, another command  can show which test suites
succeeded and which failed.

These test suites can help in many ways:

   * As discussed above, the test suites should always be run before a
     new  version is released, no matter how trivial the modifications
     to the program.

   *   If   the   software   is  internally  different  for  different
     environments  (e.g.  MPE/V vs. MPE/XL), but  should have the same
     external  behavior,  the  test  suites  should  be  run  on  both
     environments.

   *  As you're making serious changes to the software, you might want
     to  run  the test suites even before  the release, since they can
     tell you what still needs to be fixed.

   *  If you have the discipline to --  believe it or not -- write the
     test  suite before you've written your  program, you can even use
     the test suite to do the initial testing of your code. After all,
     you'd  have to initially test the  code anyway; you might as well
     use  your  test suites to do that  initial testing as well as all
     subsequent tests.

Note  also  that the test suites not only  run the program, but set up
the  proper environment for the program;  this might mean filling up a


Let's  switch  for  a moment to a  concrete example -- a date-handling
package,  something that, unfortunately, many people have had to write
on  their  own,  from  scratch.  Say that one of  the routines in your
package  is DATEADD, which adds a given  number of days to a date, and
returns  the new date. Here's the code that you might write to test it
(the dates are represented as YYYYMMDD 32-bit integers):

   IF DATEADD (19901031, 7) <> 19901107 THEN
     BEGIN
     WRITELN ('Error: DATEADD (19901031, 7) <> 19901107');
     GOT_ERROR:=TRUE;
     END;
   IF DATEADD (19901220, 20) <> 19910109 THEN
     BEGIN
     WRITELN ('Error: DATEADD (19901220, 20) <> 19910109');
     GOT_ERROR:=TRUE;
     END;
   ...

As you see, the code calls DATEADD several times, and each time checks
the result against the expected result; if the result is incorrect, it
prints  an  error  message  and sets GOT_ERROR to  TRUE. After all the
tests  are done, the program can check if GOT_ERROR is TRUE, and if it
is, say, build a special "got error" file, or write an error record to
some  special  log  record.  This  way,  the test suites  can be truly
automatic -- you can run many test suites in the background, and after
they're done, find out if all went well by just checking one file, not
looking through many large spool files for error messages.

The  first thing that you might notice  is that the DATEADD test suite
can  easily grow to be much  larger than the DATEADD procedure itself!
No  doubt  about  it  --  writing  test  suites  is  a  very expensive
proposition.   Our  test  suites  for  MPEX/3000,  SECURITY/3000,  and
VEAUDIT/3000  take  up  almost  30,000 lines,  not counting supporting
files  and  supporting  code in the actual  programs; the total source
code of our products is less than 100,000 lines. Often, writing a test
suite  for  a  feature  takes  as  long or almost  as long as actually
implementing the feature. Sometimes, instead of being reluctant to add
a  new  feature for fear of breaking  something, I am now reluctant to
add  a new feature because I don't want to bother writing a test suite
for it.

Fortunately,  the  often  dramatic  costs  of writing  test suites are
recouped  not just by the decrease in  the number of bugs, but also by
the  fact that test suites, once written,  save a lot of testing time.
It's much easier for someone to run an already-written test suite than
to execute by hand even a fraction of the tests included in the suite,
especially if they require complicated set-up. Since a typical program
will actually have to be tested several times before it finally works,
the  costs of writing a test suite  (assuming that it's written at the
same  time  as  the code, or even earlier)  can be recouped before the
program is ever released.

Also,  test suites tend to have longer  lives than code. A program can
be dramatically changed -- even re-written in another language -- and,
assuming  that it was intended to behave  the same as before, the test
suite will work every bit as well. Once the substantial up-front costs
of  writing  test  suites  have  been  paid, the pay-offs  can be very
substantial.

But  even  though we should be willing  to invest time and effort into
writing test suites, there's no reason to invest more than we have to.
In  fact,  precisely  because test suites at  first glance seem like a
luxury, and people are thus not very willing to work on them, creating
test  suites  should  be  as easy as possible. What  can we do to make
writing test suites simpler and more efficient?

One  goal that I try to shoot for is to make it as easy as possible to
add  new test cases, even if this  means doing some additional work up
front.  I try to make every new test  case, if possible, to fit on one
line.   The  reason  is  quite  simple:  I  want  to  have  as  little
disincentive  as  possible  to add new test  cases. A really fine test
suite  would  have  tests for many  different situations, including as
many obscure boundary conditions and exceptions as possible; also, any
time a new bug is found, a test should be added to the test suite that
would  have  caught  the  bug,  just  in  case the  bug re-surfaces (a
remarkably  frequent  event).  If  we  grit  our teeth  and write some
convenient  testing  tools  up  front,  we can make  it much easier to
create a full test suite.
test database, building necessary files, etc.

easily test all the features of all our products.