1. two main approaches to testing - functional or structural

   1. functional - testing based on what the system does

   2. structural - testing based on how the system works

2. specifications determine the test cases and correct results

3. functional testing

   1. black-box testing - some inputs, some outputs

   2. exhaustive test cases - through, but not optimal or usually even practical

   3. random sampling - cheap but not effective

   4. equivalence-class test cases - partition exhaustive test cases into equivalent sets of test cases

      1. some test cases should have the same result, so just using one should work

      2. the spec determines which test cases should have the same result

      3. the obvious partitioning - valid and invalid results

      4. further conditions can further partition equivalence classes

      5. output equivalence classes too - combinations of possible outputs

   5. boundary values - values at the edges of equivalence classes

      1. output boundary values too

   6. cause and effect graphing - trying to fill in the black box

      1. causes characterize inputs, effects characterize outputs

      2. the spec establishes the connection between cause and effects

      3. the resulting graph serves as a driver for test case generation

   7. special cases and experience

      1. the spec defines special cases

      2. standard programming (mal) practice defines special cases - off by one errors, accumulating garbage in c++ programs

      3. experience suggests others - unchecked text input

4. structural testing

   1. white- (or clear- or glass-) box testing - inputs, outputs, and what's between

   2. the effort is on testing what's in-between the inputs and outputs

   3. execute every statement in the module

   4. control-flow graph - all the paths from entrance to exit

   5. branch coverage - take each branch in each direction

      1. complex conditionals hide much information - require each condition component to be true and false

   6. execute all paths from the entrance to the exit - loops are the problem here

      1. use cyclomatic complexity to bound the loop iterations

   7. data-flow testing

      1. variable def-use - use in computation or predicate

      2. all-defs test - execution uses every variable value

      3. all-p-uses - execution uses every conditional variable value

      4. all-p-uses, some-c-uses

      5. all-uses

   8. mutation testing

      1. seeding errors and looking for them

      2. create program mutations by changing the program under test

         1. mutations may change operators, or add one to array indices

         2. classes of mutations

      3. mutation testing steps

         1. generate the test cases for P

         2. generate a set of mutants for P

         3. run the test cases over the mutant set

         4. each undetected mutant is live

         5. augment the test cases to detect the live mutants

      4. difficulties with mutation analysis

         1. finding live mutants equivalent to P

         2. augmenting the test set to detect live mutants

         3. determining what the improved test set means to P

         4. determining the mutant set

         5. running the mutation tests

      5. difficult but worth it

   9. support for structural testing

      1. automated support is a natural, but many problems are intractable or uncomputable

      2. generating test data

      3. running the tests

      4. analyzing the test results

      5. characterizing the result quality

This page last modified on 24 March 2000.