1. testing as an activity itself, instead of an add-on

2. testing is dynamic analysis - running programs over data, getting output

3. divide and conquer and re-assemble (or integrate)

4. basics

   1. error, fault, failure

      1. error - a program corruption (noun), a human activity (verb)

      2. faults - the condition that results from executing an error

      3. failures - the external manifestation of a fault; observation

      4. errors imply faults, but faults do not necessarily imply failures

      5. it is only in their observation that faults imply failures

         1. the absence of observable failures does not imply the absence of faults or errors

         2. testing can show the presence of errors, never their absence

      6. finding failures - testing

      7. finding faults - debugging

      8. finding errors - maintenance

   2. what is correctness

      1. correctness is expected behavior

      2. oracles establish and measure expected behavior

      3. specification provide the basis for establishing correctness

      4. human or automated oracles

         1. automated oracles good at measuring correctness

         2. human oracles are good at establishing correctness

   3. top-down and bottom-up testing

      1. it is difficult to cover failures in large programs

      2. it is difficult to find faults and errors in large programs

      3. divide and conquer

      4. divide into a hierarcy of modules

      5. test top-down; drivers

      6. test bottom-up; stubs

      7. testing should follow implementation - parallel development

   4. test cases and criteria

      1. test cases try to drive the system to faults

      2. test cases are costs - to develop, to run

      3. exhaustive test case - ideal but impractical due to costs

      4. execute all possible inputs is a testing criterion; other criteria include execute all statements, execute all paths, execute all branches

      5. specify the test criteria; generate the test cases to meet the criteria

      6. a reliable test criterion results in test cases that find faults

      7. a valid test criterion triggers every fault with a test case

      8. valid and reliable test cases result in perfect tests - which are impossible

      9. other criteria properties

         1. criteria should apply to any program and spec (applicability)

         2. structure influences test set coverage (antiextensionality)

         3. modularity influences test set coverage (antidecomposition)

         4. composition influences test set coverage (anticomposition)

         5. these properties don't guarentee good test sets, just less bad ones

         6. selecting criteria is hard

         7. selecting test cases is hard

         8. use lots of poor ones rather one good one

      10. testing psychology

          1. creativity is an important factor

          2. the soul of a sadist

          3. show programs are broken, not that they're working

          4. independent testing

This page last modified on 21 March 2000.