So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design

Experimental Economics

Volumn 14, Issue 4, 2011, Pages 439-457

  List, John A ^a,b   Sadoff, Sally ^a   Wagner, Mathis ^c  

^a UNIVERSITY OF CHICAGO   (United States)

^b NATIONAL BUREAU OF ECONOMIC RESEARCH   (United States)

^c BOSTON COLLEGE   (United States)

Author keywords

Experimental design

Indexed keywords

EID: 80255137217     PISSN: 13864157     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10683-011-9275-7     Document Type: Article

References (43)

1. Atkinson, A. C., & Donev, A. N. (1992). Optimum experimental designs. Oxford: Clarendon Press.
2. Berry, D. A. (2004). Bayesian statistics and the efficiency and ethics of clinical trials. Statistical Science, 19, 175-187.
3. Bloom, H. S. (2005). Randomizing Groups to Evaluate Place-Based Programs. In H. S. Bloom (Ed.), Learning more from social experiments: evolving analytic approaches. New York: Russell Sage Foundation.
4. Blundell, R., & Costa Dias, M. (2002). Alternative approaches to evaluation in empirical microeconomics. Portuguese Economic Journal, 1, 91-115.
5. Camerer, C. F., & Hogarth, R. M. (1999). The effects of financial incentives in experiments: a review and capital-labor-production framework. Journal of Risk and Uncertainty, 19, 7-42.
6. Cochran, W. G., & Cox, G. M. (1950). Experimental designs. New York: Wiley.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale: Erlbaum Associates.
8. Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization in health research. London: Holder Arnold.
9. Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: a toolkit (CEPR Discussion Paper No. 6059). Center for Economic Policy Research, London, England.
10. El-Gamal, M. A., & Palfrey, T. R. (1996). Economical experiments: Bayesian efficient experimental designs. International Journal of Game Theory, 25, 495-517.
11. El-Gamal, M. A., McKelvey, R. D., & Palfrey, T. R. (1993). A Bayesian sequential experimental study of learning in games. Journal of the American Statistical Association, 88, 428-435.
12. Fisher, R. A. (1935). The design of experiments. Edinburgh: Oliver and Boyd.
13. Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions. Hoboken: Wiley-Interscience.
14. Greenwald, A. G. (1976). Within-subjects design: to use or not to use. Psychological Bulletin, 83, 314-320.
15. Hahn, J., Hirano, K., & Karlan, D. (2011). Adaptive experimental design using the propensity score. Journal of Business and Economic Statistics, 29(1), 96-108.
16. Harrison, G., & List, J. A. (2004). Field experiments. Journal of Economic Literature, XLII, 1009-1055.
17. Harrison, G. W., Lau, M. I., & Rutström, E. E. (2009). Risk attitudes, randomization to treatment, and self-selection into experiments. Journal of Economic Behavior & Organization, 70, 498-507.
18. Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-960.
19. Kanninen, B. J. (2002). Optimal design for multinomial choice experiments. Journal of Marketing Research, 39(2), 14-227.
20. Karlan, D., & List, J. A. (2007). Does price matter in charitable giving? Evidence from a large-scale natural field experiment. The American Economic Review, 97, 1774-1793.
21. Keren, G. (1993). Between or within-subjects design: a methodological dilemma. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: methodological issues. Hillsdale: Erlbaum Associates.
22. Kish, L. (1965). Survey sampling. New York: Wiley.
23. Kramer, M. S., & Shapiro, S. H. (1984). Scientific challenges in the application of randomized trials. Journal of the American Medical Association, 252(19), 2739-2745.
24. Lenth, R. V. (2001). Some practical guidelines for effective sample size determination. The American Statistician, 55, 187-193.
25. Lenth, R. V. (2006-2009). Java applets for power and sample size [Computer Software]. http://www. stat. uiowa. edu/rlenth/Power. Accessed February 2010.
26. Levitt, S. D., & List, J. A. (2009). Field experiments in economics: the past, the present, and the future. European Economic Review, 53(1), 1-18.
27. Levitt, S. D., & List, J. A. (2007). What do laboratory experiments measuring social preferences tell us about the real world? The Journal of Economic Perspectives, 21(2), 153-174.
28. List, J. A. (2001). Do explicit warnings eliminate the hypothetical bias in elicitation procedures? Evidence from field auctions for sportscards. The American Economic Review, 91(5), 1498-1507.
29. List, J. A. (2006). Field experiments: a bridge between lab and naturally occurring data. Advances in Economic Analysis & Policy, 6(2), 8.
30. Liu, X., Spybrook, J., Congden, R., Martinez, A., & Raudenbush, S. W. (2009). Optimal design for longitudinal and multilevel research v. 2. 0 [Computer Software]. Available at: http://www. wtgrantfoundation. org/resources/overview/research_tools.
31. Loomes, G. (2005). Modeling the stochastic component of behaviour in experiments: some issues for the interpretation of data. Experimental Economics, 8, 301-323.
32. Mead, R. (1988). The design of experiments: statistical principles for practical application. Cambridge: Cambridge University Press.
33. McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2(1), 3-19.
34. Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173-185.
35. Raudenbush, S. W., Martinez, A., & Spybrook, J. (2007). Strategies for improving precision in group-randomized experiments. Educational Evaluation and Policy Analysis, 29(1), 5-29.
36. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies of causal effects. Biometrika, 70, 41-55.
37. Rubin, D. B. (1978). Bayesian inference for causal effects: the role of randomization. Annals of Statistics, 6, 34-58.
38. Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: a methodological inquiry and experimental test. Games and Economic Behavior, 67, 616-632.
39. Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15, 657-680.
40. Spybrook, J., Raudenbush, S. W., Congden, R., & Martinez, A. (2009). Optimal design for longitudinal and multilevel research: documentation for the optimal design software (Working Paper). Available at: http://www. wtgrantfoundation. org/resources/overview/research_tools.
41. StataCorp (2007). Stata statistical software: Release 10. College Station: StataCorp LP.
42. Wilcox, R. R. (1996). Statistics for the social sciences. San Diego: Academic Press.
43. Wilcox, N. (2008). Stochastic models for binary discrete choice under risk: a critical primer and econometric comparison. In J. C. Cox & G. W. Harrison (Eds.), Research in experimental economics: Vol. 12. Risk aversion in experiments. Bingley: Emerald.