1. process metrics - the project's post mortem

   1. productivity metrics - kloc/working-month; function points too

   2. $/defect, defects/kloc

2. how reliable is the system

3. system reliability reflects testing quality

4. system reliability must be statistically modeled

5. reliability model concepts

   1. reliability - long occurrences of expected behavior

   2. the interaction of reliability and failure is fuzzy

   3. failure occurrences are unscheduled, dealt with as statistical phenomena

      1. failure probability F(t) = the probability the system has failed by time t

      2. reliability R(t) = 1 - F(t)

      3. suppose the system doesn't fail at time t - the hazard rate

      4. mean time to failure mttf

      5. failure intensity lambda(t) - the number of failures expected by time t

   4. what is time - execution or cpu time, clock or user time, or calendar time

   5. what are faults, what is faulty behavior

   6. the environment influences system reliability

6. a reliability model - musa's execution time model

   1. the inverse proportion between failure intensity and execution time

   2. as faults are found, they're fixed, increasing reliability

   3. each fix improves reliability by the same amount, a linear relation

   4. estimate the initial failure intensity and total fault count

   5. collect data, analyze statistically

   6. the clock starts at system test

   7. more faults or time to achieve expected fault intensities

   8. parameter estimation

      1. measure time between faults or faults per time

      2. collect the data, then fit curves to it - least squares

   9. do faults really get fixed, and how many of them

   10. the notion of time - execution to calendar

       1. convert execution to calendar time directly or through failures

       2. example - x person/cpu and y person/fault

       3. refine by estimating stages of the testing process

This page last modified on 4 April 2001.