Work smarter, not harder: Guidelines for designing simulation experiments

Work smarter, not harder: Guidelines for designing simulation experiments

1. Nuts and bolts

   • Terminology and Notation
     • Experimental designs indicates how to vary the settings of factors to see whether and how they affect the response
     • Factor
       • quantitative
       • -qualitative
       • Potential factors
         • distributional parameters
         • input parameters
     • Experimental units
       • Monte Carlo: single observation
       • Discrete-event stochastic: run or batch, yielding an averaged or aggregated output value.
     • Measure of interest
       • Non-terminating simulations and steady-state performance measures: care must be taken to delete data from the simulation´s warm-up period before performing the average or aggregation
       • Time: run’s output is already in the form of a single number
     • Response surface metamodels: approximate the relationship between the factors and the responses with statistical models
     • Design: matrix where every column corresponds to a factor, and the entries within the column are settings for this factor
       • design point: a particular combination of factor level is represented in each row
       • natural levels: when the row entries correspond to the actual settings that will be used
       • replication: each repetition of the whole design matriz
         • total number of experimental units: Ntot = Nb
           • b: number of replications
           • N: number of designs points
   • Pitfalls to avoid
     • 2 types of simulation studies are not well-designed experiments.
       • can occur if several people each suggest an “interesting” combination of factor settings, so a handful of design points end up being explored where many levels change simultaneously
       • can occur when people start with a “baseline” scenario and vary one factor at a time.
     • In practice simulation models easily have tens or hundreds of potential factors
     • A handful of haphazardly chosen scenarios, or a trial-and –error approac, can use up a great deal of time without addressing the fundamental question

2. Useful designs

   • What works when
     • Factorial and gridded designs: are straight forward to construct and readily explainable-even to those without statistical backgrounds.
     • Coarse grids: are most efficient if we can assume that the simulation response is well-fit by a model with only linear main effects and interactions
     • Fine grids: provide greater detail about the response and greater flexibility for constructing metamodels of the response
     • When the number of factors is large, then more efficient designs are required.
     • Resolution 5 2ˆk fractional factorials allow the linear main effects and interactions of many factos to be investigated simultaneously
   • Factorial designs
     • 2^k
       • Requieres only 2 levels for each factor
         • Can be low and high
       • Conceptually 2ˆk factorial designs, sample at the corner of a hypercube defined by the factors´ low and high settings
       • There will be k main effects, two-way interactions, three-way interactions, and so forth u to a single k-way interaction. Adding all these up yields 2ˆk-1 term plus the intercept
       • We will always need at least one degree of freedom for estimating error
       • Properties:
         • Let us examine more that one factor at a time
         • They are orthogonal designs: the pairwise correlation between any two columns is equal to zero
       • Construct:
         • Calculate the number of rows N=2ˆk
     • m^k
       • Examining each factor with any number of levels
       • Can reveal complexities
       • Scatterplot matrix of the design points shows projection of the full design onto each pair of factors
   • Latin Hypercube designs
     • Provides a flexible way of constructing efficient designs for quantitative factors
     • N >=k
       • N = replicates
       • k = number of factors
     • Work the best when most factors have many levels
   • Fractional Factorial Design
     • Efficient at identifying slopes for main effects or 2-way interactions
   • Central Composite Design
     • let the analyst estimate all full second-oirder models
     • Start with 2^k factorial or fractional factorial design. Then add a center point and two "star points" for each of the factors