Why We (Usually) Don't Have to Worry About Multiple Comparisons

Journal of Research on Educational Effectiveness

Volumn 5, Issue 2, 2012, Pages 189-211

  Gelman, Andrew ^a   Hill, Jennifer ^b   Yajima, Masanao ^c  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b NEW YORK UNIVERSITY   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Bayesian inference; hierarchical modeling; multiple comparisons; statistical significance; Type S error

Indexed keywords

EID: 84859606223     PISSN: 19345747     EISSN: 19345739     Source Type: Journal    
DOI: 10.1080/19345747.2011.618213     Document Type: Article

References (32)

1. Almond, R. G., Lewis, C., Tukey, J. W. and Yan, D. 2000. Displays for comparing a given state to many others. American Statistician, 34: 89-93.
2. Benjamini, Y. and Hochberg, Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B, 57: 289-300.
3. Benjamini, Y. and Yekutieli, D. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29: 1165-1188.
4. Efron, B. 2006. Size, power, and false discovery rates, Tech. Rep., Stanford, CA: Department of Statistics, Stanford University.
5. Efron, B. and Tibshirani, R. 2002. Empirical Bayes methods and false discovery rates for microarrays. Genetic Epidemiology, 23: 70-86.
6. Gelman, A. 2006. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis, 1: 515-533.
7. Gelman, A. 2007. Letter to the editor regarding some papers of Dr. Satoshi Kanazawa. Journal of Theoretical Biology, 245: 597-599.
8. Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. B. 2003. Bayesian data analysis, 2nd ed., London, UK: CRC Press.
9. Gelman, A. and Hill, J. 2007. Data analysis using regression and multilevel/hierarchical models, Cambridge, UK: Cambridge University Press.
10. Gelman, A. and Tuerlinckx, F. 2000. Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15: 373-390.
11. Gelman, A. and Weakliem, D. 2009. Of beauty, sex, and power: statistical challenges in estimating small effects. American Scientist, 97: 310-316.
12. Genovese, C. and Wasserman, L. 2002. Operating characteristics and extensions of the false discovery rate procedure. Journal of the Royal Statistical Society: Series B (Statistical Methodology, 64: 499-517.
13. Grant, G. R., Liu, J. and Stoeckert, C. J. 2005. A practical false discovery rate approach to identifying patterns of differential experession in microarray data. Bioinformatics, 21: 2684-2690.
14. Greenland, S. and Robins, J. M. 1991. Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Epidemiology, 2: 244-251.
15. Hsu, J. C. 1996. Multiple comparisons: Theory and methods, London, UK: Chapman and Hall.
16. Infant Health and Development Program. 1990. Enhancing the outcomes of low-birth-weight, premature infants. A multisite, randomized trial. The Infant Health and Development Program. Journal of the American Medical Association, 263: 3035-3042.
17. Johnstone, I. and Silverman, B. 2004. Needles and straw in a haystacks: Empirical Bayes approaches to thresholding a possibly sparse sequence. Annals of Statistics, 32: 1594-1649.
18. Kanazawa, S. 2007. Beautiful parents have more daughters: A further implication of the generalized Trivers-Willard hypothesis. Journal of Theoretical Biology, 244: 133-140.
19. Kane, T. J., Rockoff, J. E. and Staiger, D. O. 2007. What does certification tell us about teacher effectiveness? Evidence from New York City. Economics of Education Review, 27: 615-631.
20. Krantz, D. H. 1999. The null hypothesis testing controversy in psychology. Journal of the American Statistical Association, 94: 1372-1381.
21. Louis, T. A. 1984. Estimating a population of parameter values using Bayes and empirical Bayes methods. Journal of the American Statistical Association, 79: 393-398.
22. National Center for Education Statistics. 1997. 1996 NAEP comparisons of average scores for participating jurisdictions, Washington, DC: Government Printing Office.
23. Poole, C. 1991. Multiple comparisons? No problem!. Epidemiology, 2: 241-243.
24. Rockoff, J. 2004. The impact of individual teachers on student achievement: evidence from panel data. American Economic Review, 94: 247-252.
25. Rubin, D. B. 1981. Estimation in parallel randomized experiments. Journal of Educational Statistics, 6: 377-401.
26. Rubin, D. B., Stuart, E. A. and Zanutto, E. L. 2004. A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29: 103-116.
27. Spiegelhalter, D., Thomas, A., Best, N., Gilks, W. and Lunn, D. 2002. BUGS: Bayesian inference using Gibbs sampling, Cambridge, England: MRC Biostatistics Unit.
28. Tukey, J. W. 1953. "The problem of multiple comparisons [Mimeographed notes]". Princeton, NJ: Princeton University.
29. Wainer, H. 1986. Five pitfalls encountered when trying to compare states on their SAT scores. Journal of Educational Statistics, 11: 239-244.
30. Wainer, H. 1996. Using trilinear plots for NAEP state data. Journal of Educational Measurement, 33: 41-55.
31. Wainer, H., Hambleton, R. K. and Meara, K. 1999. Alternative displays for communicating NAEP results: A redesign and validity study. Journal of Educational Measurement, 36: 301-335.
32. Westfall, P. H. and Young, S. S. 1993. Resampling-based multiple testing: Examples and methods for p-value adjustment, New York, NY: Wiley.