Design and Analysis of Monte Carlo Studies: Improving External Validity

Abstract

As models have become more complex, mathematical derivations have become increasingly difficult or impossible. As a result, monte carlo methods have become popular to evaluate model assumption violations and their implications for unbiased estimates. However, external validity of monte carlo studies can be a significant weakness. This paper looks to explore that issue in more detail and suggests a simple design and analysis adjustment to increase the coverage of the simulation conditions which, it is argued, in turn improves the external validity of the monte carlo study. This is explored with a motivating monte carlo example based on a one variable analysis of variance.