Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

Psychonomic Bulletin and Review

Volumn 23, Issue 2, 2016, Pages 640-647

  Cramer, Angélique O J ^a   van Ravenzwaaij, Don ^b   Matzke, Dora ^a   Steingroever, Helen ^a   Wetzels, Ruud ^c   Grasman, Raoul P P P ^a   Waldorp, Lourens J ^a   Wagenmakers, Eric Jan ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF NEWCASTLE   (Australia)

^c Data Analytics   (Netherlands)

Author keywords

Benjamini Hochberg procedure; Factorial ANOVA; False discovery rate; Familywise error rate; Multiple comparison problem; Multiway ANOVA; Preregistration; Sequential Bonferroni; Type I error

Indexed keywords

ANALYSIS OF VARIANCE; HUMAN; MEDICAL RESEARCH; PSYCHOLOGY; STANDARDS; STATISTICAL ANALYSIS;

ANALYSIS OF VARIANCE; BIOMEDICAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOLOGY;

EID: 84941670134     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-015-0913-5     Document Type: Article

References (41)

1. Barber, T. X. (1976). Pitfalls in human research: Ten pivotal points. New York, NY: Pergamon Press.
2. Benjamini, Y., Drai, D., Elmer, G., Kafkafi, N., & Golani, I. (2001). Controlling the false discovery rate in behavior genetics research. Behavioural Brain Research, 125, 279–284. doi:10.1016/S0166-4328(01)00297-2
3. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57, 289–300.
4. Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29, 1165–1188.
5. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. doi:10.1038/nrn3475
6. Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610. doi:10.1016/j.cortex.2012.12.016
7. Chambers, C. D., Munafo, M., et al. (2013, 5 June). Trust in science would be improved by study pre-registration. The Guardian. Retrieved from www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration
8. de Groot, A. D. (1969). Methodology: Foundations of inference and research in the behavioral sciences. The Hague, The Netherlands: Mouton.
9. Didelez, V., Pigeot, I., & Walter, P. (2006). Modifications of the Bonferroni–Holm procedure for a multi-way ANOVA. Statistical Papers, 47, 181–209.
10. Efron, B., Storey, J., & Tibshirani, R. (2001). Microarrays, empirical Bayes methods, and false discovery rates (Technical Report, July 2001). Stanford, CA: Stanford University, Department of Statistics. Retrieved from http://statweb.stanford.edu/~ckirby/brad/papers/2001MicroEBMethods.pdf
11. Efron, B., Tibshirani, R., Storey, J. D., & Tusher, V. (2001b). Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association, 96, 1151–1160.
12. Feingold, M., & Korsog, P. E. (1986). The correlation and dependence between two F statistics with the same denominator. American Statistician, 40, 218–220.
13. Fletcher, H. J., Daw, H., & Young, J. (1989). Controlling multiple F test errors with an overall F test. Journal of Applied Behavioral Science, 25, 101–108.
14. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587–606.
15. Goldacre, B. (2009). Bad science. London, UK: Fourth Estate.
16. Hartley, H. O. (1955). Some recent developments in analysis of variance. Communications on Pure and Applied Mathematics, 8, 47–72.
17. Hochberg, Y. (1974). Some generalizations of the t-method in simultaneous inference. Journal of Multivariate Analysis, 4, 224–234.
18. Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75, 800–802.
19. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70. Retrieved from www.jstor.org/stable/4615733
20. Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75, 383–386.
21. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.
22. Klugkist, I., Post, L., Haarhuis, F., & van Wesel, F. (2014). Confirmatory methods, or huge samples, are required to obtain power for the evaluation of theories. Open Journal of Statistics, 4, 710–725.
23. Kromrey, J. D., & Dickinson, W. B. (1995). The use of an overall F test to control Type I error rates in factorial analyses of variance: Limitations and better strategies. Journal of Applied Behavioral Science, 31, 51–64.
24. Lehmann, E. L., & Romano, J. P. (2005). Generalization of the familywise error rate. Annals of Statistics, 33, 1138–1154.
25. McHugh, R. (1958). Significance level in factorial design. Journal of Experimental Education, 26, 257–260.
26. Nakagawa, S. (2004). A farewell to Bonferroni: The problems of low statistical power and publication bias. Behavioral Ecology, 15, 1044–1045.
27. Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.
28. Olejnik, S., Li, J., & Supattathum, S. (1997). Multiple testing and statistical power with modified Bonferroni procedures. Journal of Educational and Behavioral Statistics, 22, 389–406.
29. Poldrack, R. A., Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008). Guidelines for reporting an fMRI study. NeuroImage, 40, 409–414.
30. R Development Core Team. (2007). R: A language and environment for statistical computing (Version 2.15). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org
31. Rom, D. M. (1990). A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika, 77, 663–665.
32. Ryan, T. A. (1959). Multiple comparison in psychological research. Psychological Bulletin, 56, 26–47. doi:10.1037/h0042478
33. Scheffé, H. (1953). A method for judging all contrasts in the analysis of variance. Biometrika, 40, 87–110. doi:10.1093/biomet/40.1-2.87
34. Shaffer, J. P. (1986). Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association, 81, 826–831.
35. Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika, 73, 751–754.
36. Smith, R. A., Levine, T. R., Lachlan, K. A., & Fediuk, T. A. (2002). The high cost of complexity in experimental design and data analysis: Type I and Type II error rates in multiway ANOVA. Human Communication Research, 28, 515–530.
37. Tukey, J. W. (1994). The problem of multiple comparisons. In H. I. Braun (Ed.), The collected works of John W. Tukey: Vol. 8. Multiple comparisons: 1948–1983 (pp. 1–300). New York, NY: Chapman and Hall.
38. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
39. Westfall, P. H., Tobias, R. D., & Wolfinger, R. D. (2011). Multiple comparisons and multiple tests using SAS (2nd ed.). Cary, NC: SAS Institute Inc.
40. Wolfe, J. M. (2013). Registered reports and replications in Attention, Perception, & Psychophysics [Editorial]. Attention, Perception, & Psychophysics, 75, 781–783. doi:10.3758/s13414-013-0502-5
41. Wright, S. P. (1992). Adjusted p-values for simultaneous inference. Biometrics, 48, 1005–1013.