Designing simulation experiments
Use baselines, scenarios, sensitivity analysis, and repeated runs to answer a well-defined question.
On this page
- Hold the comparison rules fixed
- Use scenarios for named operational alternatives
- One-factor-at-a-time is a local diagnostic, not a general experiment design
- Match the design to the question and run budget
- Monte Carlo propagation is different from factor screening
- Keep calibration, sensitivity, and optimization separate
- Confirm promising results with fresh runs
Choose a baseline before running alternatives. Use the same run length, warm-up rule, output definitions, and analysis method across cases unless changing one of those is itself part of the experiment.
Write down the decision factor separately from background assumptions. If staffing changes together with demand, routing, and service-time assumptions, the result cannot tell you which change caused the difference.
A scenario represents a coherent alternative such as a staffing plan, routing rule, capacity expansion, or demand case. Compare it with the baseline and inspect downstream effects as well as the KPI you intended to improve.
Changing one factor while holding every other factor fixed is easy to interpret near a baseline. It can miss interactions: two factors may have little effect separately but a large effect when changed together.
Use one-factor-at-a-time for quick local checks. Use a designed experiment when several factors may interact or when simulation time is too expensive to waste on redundant combinations.
Do not choose a design because its name sounds sophisticated. Choose it from the factor types, interactions you need to estimate, response shape you expect, and the number of runs you can afford.
- Full factorial: test every selected combination; useful when the factor count and level count are small.
- Fractional factorial or screening design: use fewer combinations to identify important factors when there are many candidates.
- Latin hypercube or other space-filling design: spread runs across a continuous parameter space when the response may be nonlinear.
- Sequential design: use early results to decide where additional simulation effort is most informative.
Advanced details: design tradeoffs
- Fractional-factorial designs save runs by deliberately confounding some effects. Check which effects are aliased before interpreting a coefficient.
- Space-filling designs cover a continuous region efficiently, but coverage alone does not separate main effects from interactions.
- Sequential designs use earlier results to choose later runs. Keep exploratory adaptation separate from any final confirmatory comparison.
Monte Carlo experiments sample uncertain inputs and propagate them through the model to estimate output uncertainty or risk. They answer questions such as the probability of missing a target or the distribution of cost under uncertain demand.
Summarize tails and probabilities that matter to the decision, not only the average output.
Write down what changes before you decide what the result means.
| Procedure | What changes | What it answers |
|---|---|---|
| Replication | Random stream | How much runs vary |
| Sensitivity | Selected input | Which inputs change the result |
| Uncertainty | Uncertain inputs | How input uncertainty changes outputs |
| Calibration | Model parameters | How closely the model matches observations |
| Optimization | Decision variables | Which tested choices perform better |
Calibration asks which parameter values make the model agree with observed evidence. Sensitivity analysis asks which assumptions drive the outputs. Optimization asks which decision performs best under the model.
Do not optimize a model merely because it can be optimized. Verify and validate the model for the intended decision first, then report the uncertainty around the recommended alternative.
Exploration naturally favors alternatives that happened to look good in the runs you inspected. After narrowing the candidates, rerun the final comparison with a predeclared analysis and fresh random streams when practical.
That separation reduces the chance of reporting a lucky exploratory result as if it were a stable improvement.