Experimental Design and Analysis
Internet-based technology and survey research methods have combined to make population-based survey experiments a reality in the social sciences. Read about our experimental methods.
Related Studies
Diana C. Mutz, Robin Pemantle, Phil Pham. 2017. The Perils of Balance Testing in Experimental Design: Messy Analyses of Clean Data
Widespread concern over the credibility of published results has led to scrutiny of statistical practices. We address one aspect of this problem that stems from the use of balance tests in conjunction with experimental data. When random assignment is botched, due either to mistakes in implementation or di↵erential attrition, balance tests can be an important tool in determining whether to treat the data as observational versus experimental. Unfortunately the use of balance tests has become commonplace in analyses of “clean” data, that is, data for which random assignment can be stipulated. Here, we show that balance tests can destroy the basis on which scientific conclusions are formed, and can lead to erroneous and even fraudulent conclusions. We conclude by advocating that scientists and journal editors resist the use of balance tests in all analyses of clean data.
Diana C. Mutz, Robin Pemantle. 2015. Standards for Experimental Research: Encouraging a Better Understanding of Experimental Methods. Journal of Experimental Political Science. 2: 192-215
Diana C. Mutz. Population-Based Survey Experiments. 2011. Princeton, NJ. Princeton University Press.
Population-based survey experiments have become an invaluable tool for social scientists struggling to generalize laboratory-based results, and for survey researchers besieged by uncertainties about causality. Thanks to technological advances in recent years, experiments can now be administered to random samples of the population to which a theory applies. Yet until now, there was no self-contained resource for social scientists seeking a concise and accessible overview of this methodology, its strengths and weaknesses, and the unique challenges it poses for implementation and analysis.
Drawing on examples from across the social sciences, this book covers everything you need to know to plan, implement, and analyze the results of population-based survey experiments. But it is more than just a "how to" manual. This lively book challenges conventional wisdom about internal and external validity, showing why strong causal claims need not come at the expense of external validity, and how it is now possible to execute experiments remotely using large-scale population samples.