Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures

Journal of Development Effectiveness

Volumn 6, Issue 1, 2014, Pages 44-57

  Fink, Günther ^a   McConnell, Margaret ^a   Vollmer, Sebastian ^a,b  

^a HARVARD SCHOOL OF PUBLIC HEALTH   (United States)

^b UNIVERSITY OF GÖTTINGEN   (Germany)

Author keywords

experimental design; heterogeneous treatment effects; multiple testing; randomised field experiments

Indexed keywords

CORRECTION; EXPERIMENTAL DESIGN; STATISTICAL ANALYSIS; TESTING METHOD;

EID: 84897102181     PISSN: 19439342     EISSN: 19439407     Source Type: Journal    
DOI: 10.1080/19439342.2013.875054     Document Type: Article

References (22)

1. Angrist, J. D. 2004. "Treatment Effect Heterogeneity in Theory and Practice." The Economic Journal 114 (March): C52-C83.
2. Angrist, J., E. Bettinger, E. Bloom, E. King, and M. Kremer. 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment." American Economic Review 92 (5): 1535-1558.
3. Benjamini, Y., and Y. Hochberg. 1995. "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing." Journal of the Royal Statistical Society Series B 57 (1): 289-300.
4. Benjamini, Y., and D. Yekutieli. 2001. "The Control of the False Discovery Rate in Multiple Testing under Dependency." The Annals of Statistics 29 (4): 1165-1188. doi:10.2307/2674075.
5. Card, D., S. DellaVigna, and U. Malmendier. 2011. "The Role of Theory in Field Experiments." Journal of Economic Perspectives 25 (3): 39-62.
6. Deaton, A. 2010. "Instruments, Randomization, and Learning about Development." Journal of Economic Literature 48 424-455.
7. Duflo, E., R. Glennerster, and M. Kremer. 2008. "Using Randomization in Development Economics Research: A Toolkit." In Handbook of Development Economics, Vol. 4, edited by T. Schultz and J. Strauss, 3895-3962. Amsterdam: North Holland.
8. Engemann, K. M., and H. J. Wall. 2009. "A Journal Ranking for the Ambitious Economist." Federal Reserve Bank of St. Louis Review 91 (3): 127-139.
9. Farcomeni, A. 2008. "A Review of Modern Multiple Hypothesis Testing, with Particular Attention to the False Discovery Proportion." Statistical Methods in Medical Research 17 (4): 347-388.
10. Fisman, R., S. S. Iyengar, E. Kamenica, and I. Simonson. 2006. "Gender Differences in Mate Selection: Evidence from a Speed Dating Experiment." The Quarterly Journal of Economics 121 (2): 673-697.
11. Gelman, A., J. Hill, and M. Yajima. 2012. "Why We (Usually) Don't Worry About Multiple Comparisons." Journal of Research on Educational Effectiveness 5: 189-211.
12. Green, D. P., and H. L. Kern. 2012. "Modeling heterogeneous treatment effects in large-scale experiments using Bayesian Additive Regression Tree." Public Opinion Quarterly 76: 491-511.
13. Harrison, G. W., and J. A. List. 2004. "Field Experiments." Journal of Economic Literature 42 (4): 1009-1055.
14. Imai, K., and A. Strauss. 2011. "Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-out-the-vote Campaign." Political Analysis 19 (1): 1-19.
15. JPAL. 2011. "Abdul Latif Jameel Poverty Action Lab 2011." Accessed May 2011. http://www.povertyactionlab.org/
16. List, J. A. 2011. "Field Experiments 2011." Accessed May 2011. http://www.fieldexperiments.com/
17. Newson, R. 2003. "Multiple-test Procedures and Smile Plots." The Stata Journal 3 (2): 100-132.
18. Romano, J. P., A. M. Shaikh, and M. Wolf. 2010a. "Hypothesis Testing in Econometrics." Annual Reviews of Economics 2: 75-104.
19. Romano, J. P., A. M. Shaikh, and M. Wolf. 2010b. Multiple Testing. The New Palgrave Dictionary of Economics. Basingstoke: Palgrave Macmillan.
20. Romano, J. P., and M. Wolf. 2005. "Stepwise Multiple Testing as Formalized Data Snooping." Econometrica 73 (4): 1237-1282.
21. Simes, R. J. 1986. "An Improved Bonferroni Procedure for Multiple Tests of Significance." Biometrika 73: 751-754.
22. Westfall, P. H., and S. Stanley Young. 1993. "Resampling-based multiple testing." Statistics in Medicine 13 (10): 1084-1086. doi:10.1002/sim.4780131012.