THE DEFINITION OF TESTING
Stop for a moment to define testing for yourself. (Don't peak ahead!)
Does one of the following definition match yours?
Testing is a process designed to
• Prove that the program is error free
• Establish that the software performs its functions correctly
• Establish with confidence that the software does its job fully
If yours matches, you are in for a surprise because testing is none
of these. Rather it is properly defined as follows (see Concepts):
Concepts - Testing is the task of locating errors.
THE IMPORTANCE OF A GOOD DEFINITION
Any definition of testing is more than a passive description. It
has a profound impact on the way testing is carried out. Since people
are highly goal-oriented, the setting of proper goals can mean the
difference between success and failure. If the goal of testing were to
prove [that a program or process functions properly], the tester would
subconsciously work towards this goal, choosing test data that would
prove that point. The reverse would be true if the goal were to locate and correct
errors. Test data would be selected with an eye toward providing the
program with cases that would likely cause the program to fail. This
would be a more desirable result. Why? We begin with the assumption
that the system, like most systems, contains errors. The job of testing
is to discover them before the user does. In that case, a good tester
is one who is successful in crashing the system, or in causing it to
perform in some way that is counter to the specification.
The mentality of the tester, then, is a destructive one -quite
different from the constructive attitude of the programmer, the
"creator". This is useful information for the analyst. Who is acting as
a project leader, and is responsible for staffing. Staff should be
selected with the appropriate personality traits in mind.
Another effect of having a proper working definition of testing
regards the way the project leader assesses the performance of the test
team. Without a proper definition of testing, the leader might describe
a successful test run as one which proves the program is error free and
describe an unsuccessful test as one which found errors. As is the case
with the testers themselves, this mind-set is actually
counter-productive to the testing process. GOALS OF TESTING
To satisfy its definition, testing must accomplish the following goals:
1. Find cases where the program does not do what it is supposed to do.
2. Find cases where the program does things it is not supposed to do.
The first goal refers to specifications which were not satisfied by
the program while the second goal refers to unwanted side-effects.
THE EIGHT BASIC PRINCIPLES OF TESTING
Following are eight basic principles of testing:
1. Define the expected output or result.
2. Don't test your own programs.
3. Inspect the results of each test completely.
4. Include test cases for invalid or unexpected conditions.
5. Test the program to see if it does what it is not supposed to do as well as what it is supposed to do.
6. Avoid disposable test cases unless the program itself is disposable.
7. Do not plan tests assuming that no errors will be found.
8. The probability of locating more errors in any one module is
directly proportional to the number of errors already found in that
module.
Let's look at each of these pints.
1) DEFINE THE EXPECTED OUTPUT OR RESULT
More often that not, the tester approaches a test case without a set
of predefined and expected results. The danger in this lies in the
tendency of the eye to see what it wants to see. Without knowing the
expected result, erroneous output can easily be overlooked. This
problem can be avoided by carefully pre-defining all expected results
for each of the test cases. Sounds obvious? You’d be surprised how many
people miss this pint while doing the self-assessment test.
2) DON'T TEST YOUR OWN PROGRAMS
Programming is a constructive activity. To suddenly reverse
constructive thinking and begin the destructive process of testing is a
difficult task. The publishing business has been applying this idea for
years. Writers do not edit their own material for the simple reason
that the work is "their baby" and editing out pieces of their work can
be a very depressing job.
The attitudinal l problem is not the only consideration for this
principle. System errors can be caused by an incomplete or faulty
understanding of the original design specifications; it is likely that
the programmer would carry these misunderstandings into the test phase.
3) INSPECT THE RESULTS OF EACH TEST COMPLETELY
As obvious as it sounds, this simple principle is often overlooked.
In many test cases, an after-the-fact review of earlier test results
shows that errors were present but overlooked because no one took the
time to study the results.
4) INCLUDE TEST CASES FOR INVALID OR UNEXPECTED CONDITIONS
Programs already in production often cause errors when used in some
new or novel fashion. This stems from the natural tendency to
concentrate on valid and expected input conditions during a testing
cycle. When we use invalid or unexpected input conditions, the
likelihood of boosting the error detection rate is significantly
increased.
5) TEST THE PROGRAM TO SEE IF IT DOES WHAT IT IS NOT SUPPOSED TO DO AS WELL AS WHAT IT IS SUPPOSED TO DO
It's not enough to check if the test produced the expected output.
New systems, and especially new modifications, often produce unintended
side effects such as unwanted disk files or destroyed records. A
thorough examination of data structures, reports, and other output can
often show that a program is doing what is not supposed to do and
therefore still contains errors.
6) AVOID DISPOSABLE TEST CASES UNLESS THE PROGRAM ITSELF IS DISPOSABLE
Test cases should be documented so they can be reproduced. With a
non-structured approach to testing, test cases are often created
on-the-fly. The tester sits at a terminal, generates test input, and
submits them to the program. The test data simply disappears when the
test is complete.
Reproducible test cases become important later when a program is
revised, due to the discovery of bugs or because the user requests new
options. In such cases, the revised program can be put through the same
extensive tests that were used for the original version. Without saved
test cases, the temptation is strong to test only the logic handled by
the modifications. This is unsatisfactory because changes which fix one
problem often create a host of other apparently unrelated problems
elsewhere in the system. As considerable time and effort are spent in
creating meaningful tests, tests which are not documented or cannot be
duplicated should be avoided.
7) DO NOT PLAN TESTS ASSUMING THAT NO ERRORS WILL BE FOUND
Testing should be viewed as a process that locates errors and not
one that proves the program works correctly. The reasons for this were
discussed earlier.
8) THE PROBABILITY OF LOCATING MORE ERRORS IN ANY ONE MODULE IS
DIRECTLY PROPORTIONAL TO THE NUMBER OF ERRORS ALREADY FOUND IN THAT
MODULE
At first glance, this may seem surprising. However, it has been
shown that if certain modules or sections of code contain a high number
of errors, subsequent testing will discover more errors in that
particular section that in other sections.
Consider a program that consists of two modules, A and B. If testing
reveals five errors in module A and only one error in module B, module
A will likely display more errors that module B in any subsequent tests.
Why is this so? There is no definitive explanation, but it is
probably due to the fact that the error-prone module is inherently
complex or was badly programmed. By identifying the most "bug-prone"
modules, the tester can concentrate efforts there and achieve a higher
rate of error detection that if all portions of the system were given
equal attention.
Extensive testing of the system after modifications have been made is referred to as regression testing.
|