Lecture Notes for Simulation

18 April 2005 - Statistical Experiment Design

Outline

A Problem

• A two-server, one-queue system.

• A device bolts-on to a server and improves (reducdes) service time by 10%.

• How many devices should bought, and what's the effect of doing so?

  • The second question is open ended.

• These kinds of questions can be thought of as optimization questions.

  • The general question is: What's the best possible outcome?

Terminology

A responsive variable is a simulation output of interest in the experiment.
• A factor is a simulation input or parameter that influences a responsive variable.

• A factor's levels are the acceptable values that may be assigned to the factor.

• A treatment assigns a particular level to a factor.

• Replicating an experiment repeatedly runs the simulation with the same treatment set.

• Not all simulation inputs, parameters, or outputs need to be accounted for as factors or responsive variables.

  • The trick is figuring out which ones are.

Randomized Design Experiments

• The simplest case has a single factor influencing a single responsive variable.

• The simplest question to ask is "Is there any difference?".

  • Not "What is..." or "How much...".

• A completely randomized design experiment (or just randomized design) uses the simplest case to answer the simplest question.

Randomized Design Procedure

• Given a factor f has levels { l₁, l₂, ..., l_k }, running a randomized-design experiment involves:

  1. Forming a random permutation of the levels (l_p1, l_p2, ..., l_pk).

  2. Runing the simulation k times, assigning l_pi to f in the ith simulation.

  3. Repeating steps 1 and 2 for as many replications as desired.

• Random level assignment averages out the effects of other simulation inputs and parameters on the responsive variables.

  • That's the hope, anyway.

Randomized Design Model

• A randomized-design experiment with t treatments and r repetitions results in t*r samples.

  • The sample Y_ij is the result of the ith run of the simulation using the jth level.

  • Y_ij is the mean of samples drawn from the responsive variable.

• A randomized design models a sample Y_ij as
  Y_ij = Y + T_j + e_ij

  where

  Y is the sample mean, sum(i = 1 to t, sum(j = 1 to l, Y_ij))/t*l.

  T_j is the cumulative effect of treatment j. T = sum(i = 1 to t, Y_ij).

  e_ij is the random error associated with sample Y_ij.

Analysis of Variance

• Questions about the optimum rely on relative and not absolute analysis.

• Optimum questions shift attention away from the mean and towards the variance.

  • The mean is an absolute but probably wrong value.

  • The variance is a relative measure indicating how wrong the mean may be.

• Analysis of variance (ANOVA) is a statistical analysis of variance.

Randomized Design Data Analysis

• The sample model Y_ij = Y + T_j + e_ij is represented statistically by
  SS_total = SS_mean + SS_trmt + SS_error

  where

  SStotal	=	sum(i = 1 to t, sum(j = 1 to l, Yij2))
  SSmean	=	NY**2
  SStrmt	=	sum(i = 1 to t, ni(Y*i - Y**)2)
  SSerr	=	sum(i = 1 to t, sum(j = 1 to l, (Yij - Y*j)2))

  and come from ANOVA, and

  Y*j	=	sum(i = 1 to t, Yij)/nj
  Y**	=	sum(i = 1 to t, sum(j = 1 to nj, Yij))/N
  nj	=	level j run count
  N	=	sum(i = 1 to l, ni)

Hypothesis Testing

• The question (remeber that?) was: Does the device have any effect?

• This question corresponds to the hyphotheses
  H₀: for all levels i, T_i = 0.

  H_a: for some level i, T_i != 0.

• The test statistic is the ratio of the treatment variance over the error variance:
  F = (SS_trmt/(l - 1))/(SS_err/(N - l)

• F is a sample from an F distribution with two degrees of freedom (l - 1 and N - l).

• For a given confidence level a, reject H₀ if F > F(a, l - 1, N - l).

Experiment Example

• The server-queue example has four treatments depending on the number of devices used in an experiment

  • The levels are labeled 0 through 3.

• Three replications of the expeiment require three random permutations of the set { 0, 1, 2, 3 }.

• Three experiments produce the following samples:

  experiments	0	1	2	3
  1	50	48	61	58
  2	32	18	64	72
  3	80	73	54	63

Analysis Example

• ANOVA over the experiment samples produces

  SStotal	=	SSmean	+	SStrmt	+	SSerr
  41,131	=	37,739.6	+	902.17	+	2,489.23

• The associated F value is
  (SS_trmt/(l - 1))/(SS_err/(N - l) = (902.17/3)/(2489.23/8) = 0.96.

• At a 95% confidence level, F(0.95, 3, 8) = 4.07 and H₀ can't be rejected; that is, the devices may have had an effect.

The Design Revisited

• Complete randomized design ignored the queue arrival rate.

  • Either it has no siginificant effect, or

  • The significant effects average to zero.

• Sometimes these assumptions are valid (in an assembly line) and some times they're not (in a bank).

• When non-factor inputs and prameters can't be assumed away, they must be accounted for in the experiment.

• The idea is to group all significant non-factor influences into a block and account for that block's effects in data analysis.

Randomized Block Design

• Randomized block design is similar to complete randomized design but adds a term to account for non-factor influences.

  • The term is called a block.

• The block is essentially another factor in the experiment.

  • It has its own level set, it influences the experiments run, and it participates in analysis.

Points to Remember

This page last modified on 22 April 2005.