1. two main approaches to testing - functional or structural

   1. functional - testing based on what the system does

   2. structural - testing based on how the system works

2. specifications determine the test cases and correct results

3. functional testing

   1. black-box testing - some inputs, some outputs

   2. exhaustive test cases - through, but not optimal or usually even practical

   3. random sampling - cheap but not effective

   4. equivalence-class test cases - partition exhaustive test cases into equivalent sets of test cases

      1. some test cases should have the same result, so just using one should work

      2. the spec determines which test cases should have the same result

      3. the obvious partitioning - valid and invalid results

      4. further conditions can further partition equivalence classes

      5. output equivalence classes too - combinations of possible outputs

   5. boundary values - values at the edges of equivalence classes

      1. output boundary values too

   6. cause and effect graphing - trying to fill in the black box

      1. causes characterize inputs, effects characterize outputs

      2. the spec establishes the connection between cause and effects

      3. the resulting graph serves as a driver for test case generation

   7. special cases and experience

      1. the spec defines special cases

      2. standard programming (mal) practice defines special cases - off by one errors, accumulating garbage in c++ programs

      3. experience suggests others - unchecked text input

4. structural testing

   1. white- (or clear- or glass-) box testing - inputs, outputs, and what's between

   2. the effort is on testing what's in-between the inputs and outputs

   3. execute every statement in the module

   4. control-flow graph - all the paths from entrance to exit

   5. branch coverage - take each branch in each direction

      1. complex conditionals hide much information - require each condition component to be true and false

   6. execute all paths from the entrance to the exit - loops are the problem here

      1. use cyclomatic complexity to bound the loop iterations

   7. data-flow testing

      1. variable def-use - use in computation or predicate

      2. all-defs test - execution uses every variable value

      3. all-p-uses - execution uses every conditional variable value

      4. all-p-uses, some-c-uses

      5. all-uses

   8. mutation testing

      1. seeding errors and looking for them

      2. create program mutations by changing the program under test

         1. mutations may change operators, or add one to array indices

         2. classes of mutations

      3. mutation testing steps

         1. generate the test cases for P

         2. generate a set of mutants for P

         3. run the test cases over the mutant set

         4. each undetected mutant is live

         5. augment the test cases to detect the live mutants

      4. difficulties with mutation analysis

         1. finding live mutants equivalent to P

         2. augmenting the test set to detect live mutants

         3. determining what the improved test set means to P

         4. determining the mutant set

         5. running the mutation tests

      5. difficult but worth it

   9. support for structural testing

      1. automated support is a natural, but many problems are intractable or uncomputable

      2. generating test data

      3. running the tests

      4. analyzing the test results

      5. characterizing the result quality

5. testing object-oriented systems.

   1. functional testing still applies to objects and methods

   2. structural testing still applies to methods

   3. translucent box testing - structural testing for objects.

      1. testing method and state interaction; not method-only testing

   4. adt testing provides a framework, but inheritance and a less formal orientation

   5. issues

      1. classes and instances - abstract classes

      2. object state is difficult to characterize - non-sequential control flow, global state influences

      3. inheritance hierarchy

         1. some of the testable parts aren't in the class

         2. some of the previously tested components aren't the same

         3. arbitrary patterns of method invocation

   6. state-based (translucent box) testing

      1. characterizing state changes induced by method calls

      2. invoke every method in every state

         1. potentially expensive - state may be large

         2. one invocation per state may not be enough

      3. reduce the state space using equivalence classes

      4. re-introduce special cases

      5. also check the integrity of the data structures making up the state

      6. testing is fairly standard - test harness, special set and examine methods, stubbing out called objects

      7. comments

         1. testing constructors and destructors is difficult

         2. testing effort related to state space

         3. errors not related to state abuse are not detected

         4. establishing a testing order among objects may be hard - bottom up

   7. incrementally testing objects

      1. can the tests in a parent class be reused in a child class

      2. avoid redundant testing, reduce the number of test cases

      3. new, old (recursive), and redefined features

      4. feature interaction between parent and child

      5. select test cases to tickle inter-feature interaction

      6. test-case criteria based on inter- and intra feature interaction patterns

      7. top-down testing

This page last modified on 28 March 2001.