Common random numbers for scenario comparison
Pair random inputs across alternatives to reduce noise in a comparison, then check whether the pairing actually helped.
Suppose you compare three and four service desks. With independent random inputs, one alternative may receive an easier set of arrivals simply by chance.
Common random numbers give the alternatives matching random inputs. The goal is to make the A/B difference less noisy so the effect of the changed decision is easier to estimate.
For replication 1, run A and B with the paired random streams and record B − A. Repeat that pairing for replication 2, replication 3, and so on. Estimate the mean and confidence interval of those differences.
paired effect for run i = result(Bᵢ) − result(Aᵢ)
Common random numbers help when the pairing makes the two responses positively related. They can fail to reduce variance, and some systems can even become noisier.
Keep separate random streams for different sources of randomness so the intended pairing does not disappear when one part of a model consumes an extra random draw.
Advanced details: why positive pairing helps
For a paired difference B minus A, positive covariance between A and B reduces the variance of the difference. If the pairing creates little correlation, there may be little benefit; if it creates negative correlation, the comparison can become noisier.
Var(B - A) = Var(B) + Var(A) - 2 Cov(A, B)