CS 477 Lecture Notes
Week Three, Tuesday: Testing Principles
Handout: "Risks of Content Filtering," Communications of the ACM
Guest speaker: Luigi Manna, USC CS Department System Administrator
Quality Issues
The term "V&V" is commonly (wrongly) used to denote testing. The old fashioned
(wrong) process model lumps testing into a single phase. "Verification" means determining
if a phase has been performed properly. "Validation" refers to an acceptance process.
Boehm's definitions (which I prefer) are:
Verification: Are we building the product right?
Validation: Are we building the right product?
Quality: the extent to which the product satisfies its specifications.
In some organizations, the role of SQA is really that of "process police."
They ensure that the development (and maintenance) team is following the
right process. If the team skips some step, SQA raises a flag and halts
the process.
Figure 1: Testing should not be a separate phase.
Figure 2: Testing should be distributed throughout development and maintenance.
Nonexecution-Based Testing
Document walkthrough: advance document review--list of not understood items and list of
incorrect items. Record faults only.
Document inspection: more intensive than a walkthrough with 5 formal steps.
Checklist of potential faults. Written report by inspection leader.
bug = fault
error => fault => failure
Execution-Based Testing
Dijkstra: Execution testing can never prove a program is bug-free.
What Should Be Tested?
Input space must always be a small subset of potential inputs.
Test a real-time system with a simulator.
Utility, reliability, robustness, performance, correctness.
Testing versus Correctness Proofs
Correctness proofs can supplement testing when the cost is justified.
Who Should Perform Execution-Based Testing?
Systematic testing should not be performed by the programmer.
When Testing Stops
Testing stops when the product has been removed from service
(because it becomes easier to build new than to maintain an old system, for example).
This page established January 30, 1998; last updated January 24, 2000.