Planned hypothesis tests are not necessarily exempt from multiplicity adjustment

Journal of Research Practice

Volumn 11, Issue 1, 2015, Pages

  Frane, Andrew V ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Hypothesis testing; Null hypothesis; Statistical inference; Statistical methods; Type I error

Indexed keywords

EID: 84944472740     PISSN: None     EISSN: 1712851X     Source Type: Journal    
DOI: None     Document Type: Article

References (83)

1. Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Associates.
2. Aickin, M. (1999). Other method for adjustment of multiple testing exists. BMJ, 318(7176), 127-128.
3. American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.). Washington, D.C.: Author.
4. Anderson, N. H. (2014). Empirical direction in design and analysis. New York, NY: Routledge.
5. Armstrong, R. A. (2014). When to use the Bonferroni correction. Ophthalmic and Physiological Optics, 34(5), 502-508.
6. Askari, S., Kirby, R. L., Parker, K., Thompson, K., & O'Neill, J. (2013). Wheelchair propulsion test: Development and measurement properties of a new test for manual wheelchair users. Archives of Physical Medicine and Rehabilitation, 94(9), 1690-1698.
7. Baguley, T. S. (2012). Serious stats: A guide to advanced statistics. New York, NY: Palgrave Macmillan.
8. Bechofer, R. E., & Dunnett, C. W. (1982). Multiple comparisons for orthogonal contrasts: Examples and tables. Technometrics, 24(3), 213-222.
9. Bender, R., & Lange, S. (1998). What's wrong with arguments against multiplicity adjustments [Letter to the editor]. BMJ. Retrieved from http://www.bmj.com/rapid-response/2011/10/27/whats-wrong-arguments-against-multiplicity-adjustments
10. Bender, R., & Lange, S. (2001). Adjusting for multiple testing-when and how? Journal of Clinical Epidemiology, 54(4), 343-349.
11. Benjamini, Y. (2010). Simultaneous and selective inference: Current successes and future challenges. Biometrical Journal, 52(6), 708-721.
12. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B, 57(1), 289-300.
13. Benjamini, Y., & Yekutieli, D. (2005). False discovery rate: Adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association, 100(469), 71-81.
14. Bretz, F., & Westfall, P. H. (2014). Multiplicity and replicability: Two sides of the same coin. Pharmaceutical Statistics, 13(6), 343-344.
15. Brown, S. R. (1990). Experimental design and analysis. Newbury Park, CA: Sage.
16. Cachelin, F. M., Shea, M., Phimphasone, P., Wilson, G. T., Thompson, D. R., & Striegel, R. H. (2014). Culturally adapted cognitive behavioral guided self-help for binge eating: A feasibility study with Mexican Americans. Cultural Diversity and Ethnic Minority Psychology, 20(3), 449-457.
17. Clifford, H. D., Hayden, C. M., Khoo, S., Zhang, G., Le Souëf, P. N., & Richmond, P. (2012). CD46 measles virus receptor polymorphisms influence receptor protein expression and primary measles vaccine responses in naive Australian children. Clinical and Vaccine Immunology, 19(5), 704-710.
18. Cohen, B. H. (2013). Explaining psychological statistics (4th ed.). Hoboken, NJ: John Wiley.
19. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
20. Dmitrienko, A., Tamhane, A. C., & Bretz, F. (Eds.). (2010). Multiple testing problems in pharmaceutical statistics. Boca Raton, FL: Chapman & Hall.
21. Doncaster, C. P., & Davey, A. J. H. (2007). Analysis of variance and covariance: How to choose and construct models for the life sciences. Cambridge, UK: Cambridge University Press.
22. Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52-64.
23. European Agency for the Evaluation of Medicinal Products. (2002). Points to consider on multiplicity issues in clinical trials. Retrieved from http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003640.pdf
24. Fenesi, B., Heisz, J. J., Savage, P. I., Shore, D. I., & Kim, J. A. (2014). Combining best-practice and experimental approaches: Redundancy, images, and misperceptions in multimedia learning. The Journal of Experimental Education, 82(2), 253-263.
25. Finner, H., & Roters, M. (2001). On the false discovery rate and expected Type I errors. Biometrical Journal, 43(8), 985-1005.
26. Glaus, J., Vandeleur, C. L., von Känel, R., Lasserre, A. M., Strippoli, M. F., Gholam-Rezaee, M., Preisig, M. (2014). Associations between mood, anxiety or substance use disorders and inflammatory markers after adjustment for multiple covariates in a population-based study. Journal of Psychiatric Research, 58, 36-45.
27. Glickman, M. E., Rao, S. R., & Schultz, M. R. (2014). False discovery rate control is a recommended alternative to Bonferroni-type adjustments in health studies. Journal of Clinical Epidemiology, 67(8), 850-857.
28. Goeman, J. J., & Solari, A. (2014). Multiple hypothesis testing in genomics. Statistics in Medicine, 33(11), 1946-1978.
29. Ha, R. R., & Ha, J. C. (2012). Integrative statistics for the social and behavioral sciences. Thousand Oaks, CA: Sage.
30. Harkness, K. L., & Luther, J. (2001). Clinical risk factors for the generation of life events in major depression. Journal of Abnormal Psychology, 110(4), 564-572.
31. Hays, W. L. (1988). Statistics (4th ed.). Fort Worth, TX: Holt, Rinehart and Winston.
32. Helweg-Larsen, M., & Nielsen, G. A. (2009). Smoking cross-culturally: Risk perceptions among young adults in Denmark and the United States. Psychology and Health, 24(1), 81-93.
33. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.
34. Holmes, A. L., Joyce, K., Xie, H., Falank, C., Hinz, J. M., & Wise, J. P., Sr. (2014). The impact of homologous recombination repair deficiency on depleted uranium clastogenicity in Chinese hamster ovary cells: XRCC3 protects cells from chromosome aberrations, but increases chromosome fragmentation. Mutation Research: Fundamental and Molecular Mechanisms of Mutagenesis, 762, 1-9.
35. Hommel, G. (1988). A stagewise multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383-386.
36. Howell, D. C. (2013). Statistical methods for psychology (8th ed.). Belmont, CA: Wadsworth.
37. Hsu, J. C. (1996). Multiple comparisons. Boca Raton, FL: Chapman & Hall.
38. Iacobucci, D. (2001). Analysis of variance: I.A. Can I test for simple effects in the presence of an insignificant interaction? Journal of Consumer Psychology, 10(1), 5-9.
39. Kawai, V. K., Avalos, I., Oeser, A., Oates, J. A., Milne, G. L., Solus, J., Stein, C. M. (2014). Suboptimal inhibition of platelet cyclooxygenase 1 by aspirin in systemic lupus erythematosus: Association with metabolic syndrome. Arthritis Care and Research, 66(2), 285-292.
40. Keppel, G., & Zedeck, S. (1989). Data analysis for research designs. New York, NY: W. H. Freeman.
41. Kirk, R. E. (2013). Experimental design: Procedures for the behavioral sciences (4th ed.). Thousand Oaks, CA: Sage.
42. Krane-Gartiser, K., Henriksen, T. E. G., Morken, G., Vaaler, A., & Fasmer, O. B. (2014). Actigraphic assessment of motor activity in acutely admitted inpatients with bipolar disorder. PLoS ONE, 9(2), e89574.
43. Lau, M., Lin, H., & Flores, G. (2012). Racial/ethnic disparities in health and health care among U.S. adolescents. Health Services Research, 47(5), 2031-2059.
44. MacDonald, B., & Barry, R. J. (2014). Trial effects in single-trial ERP components and autonomic responses at very long ISIs. International Journal of Psychophysiology, 92(3), 99-112.
45. McHugh, R. B., & Ellis, D. S. (1957). The 'post-mortem' testing of experimental comparisons. Psychological Bulletin, 52(5), 425-428.
46. McKillup, S. (2012). Statistics explained: An introductory guide for life scientists. Cambridge, UK: Cambridge University Press.
47. Miller, R. G. (1981). Simultaneous statistical inference (2nd ed.). New York, NY: Springer-Verlag.
48. Nam, C. W., & Zellner, R. D. (2011). The relative effects of positive interdependence and group processing on student achievement and attitude in online cooperative learning. Computers and Education, 56(3), 680-688.
49. Nieuwenhuis, I. L. C., Folia, V., Forkstam, C., Jensen, O., & Petersson, K. M. (2013). Sleep promotes the extraction of grammatical rules. PLoS ONE, 8(6), e65046.
50. O'Keefe, D. J. (2003). Colloquy: Should familywise alpha be adjusted? Against familywise alpha level adjustment. Human Communication Research, 29(3), 431-447.
51. Pagano, R. R. (2013). Understanding statistics in the behavioral sciences (10th ed.). Boston, MA: Cengage.
52. Pashler, H., & Wagenmakers, E. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528-530. Retrieved from http://pps.sagepub.com/content/7/6/528.full.pdf
53. Pataki, K. W., Metz, A. E., & Pakulski, L. (2014). The effect of thematically related play on engagement in storybook reading in children with hearing loss. Journal of Early Childhood Literacy, 14(2), 240-264.
54. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design and analysis: An integrated approach. New York, NY: Psychology.
55. Perneger, T. V. (1998). What's wrong with Bonferroni adjustments. BMJ, 316(7139), 1236-1238.
56. Pyra, M., Weber, K., Wilson, T. E., Cohen, J., Murchison, L., Goparaju, L., & Cohen, M. H. (2014). Sexual minority status and violence among HIV-infected and at-risk women. Journal of General Internal Medicine, 29(8), 1131-1138.
57. Randolph, K. A., & Meyers, L. L. (2013). Basic statistics in multivariate analysis. Oxford, England: Oxford University Press.
58. Roche, K., & Chainay, H. (2013). Visually guided grasping of common objects: Effects of priming. Visual Cognition, 21(8), 1010-1032.
59. Rovai, A. P., Baker, J. D., & Ponton, M. K. (2014). Social science research design and statistics: A practitioner's guide to research methods and IBM SPSS analysis. Chesapeake, VA: Waterfree.
60. Rutherford, A. (2011). ANOVA and ANCOVA: A GLM approach. Hoboken, NJ: John Wiley.
61. Ryan, T. A. (1959). Multiple comparisons in psychological research. Psychological Bulletin, 56(1), 26-47.
62. Ryan, T. A. (1962). The experiment as the unit for computing rates of error. Psychological Bulletin, 59(4), 301-305.
63. Ryan, T. A. (1995, December 1). 'Post hoc' tests [posted to STAT-L discussion group]. Retrieved from http://groups.google.com/forum/#!activity/sci.stat.consult/JznJOuHfr7QJ/sci.stat.consult/fJmYD9TJQ6A/AfMk-8gvJacJ
64. Sheskin, D. (2011). Handbook of parametric and nonparametric statistical procedures (5th ed.). Boca Raton, FL: Chapman & Hall.
65. Sijbrandij, M., Engelhard, I. M., Lommen, M. J. J., Leer, A., & Baas, J. M. P. (2013). Impaired fear inhibition learning predicts the persistence of symptoms of posttraumatic stress disorder (PTSD). Journal of Psychiatric Research, 47(12), 1991-1997.
66. Stangor, C. (2015). Research methods for the behavioral sciences (5th ed.). Stamford, CT: Cengage.
67. Stanley, J. C. (1957). Additional 'post-mortem' tests of experimental comparisons. Psychological Bulletin, 54(2), 128-130.
68. Steinfatt, T. M. (2006). The alpha percentage and experimentwise error rates in communication research. Human Communication Research, 5(4), 366-374.
69. Stenfors, C. U. D., Marklund, P., Hanson, L. L. M., Theorell, T., & Nilsson, L. (2014). Are subjective cognitive complaints related to memory functioning in the working population? BMC Psychology, 2(3), 1-14. Retrieved from http://www.biomedcentral.com/content/pdf/2050-7283-2-3.pdf
70. Streiner, D. L. (2015). Best (but oft-forgotten) practices: The multiple problems of multiplicity-whether and how to correct for many statistical tests. American Journal of Clinical Nutrition. Advance online publication.
71. Streiner, D. L, & Norman, G. R. (2011). Correction for multiple testing: Is there a resolution? Chest, 140(1), 16-18.
72. Tabachnick, B. G., & Fidell, L. S. (2012). Using multivariate statistics (6th ed.). Boston, MA: Pearson.
73. Tamhane, A. C. (2009). Statistical analysis of designed experiments: Theory and applications. Hoboken, NJ: John Wiley.
74. Thompson, B. (1994). Planned versus unplanned and orthogonal versus nonorthogonal contrasts: The neo-classical perspective. In B. Thompson (Ed.), Advances in social science methodology (Vol. 3, pp. 3-27). Greenwich, CT: JAI.
75. Tromovitch, P. (2012). Statistical reporting with Philip's sextuple and extended sextuple: A simple method for easy communication of findings. Journal of Research Practice, 8(1), Article P2. Retrieved from http://jrp.icaap.org/index.php/jrp/article/view/323/270
76. Tucker, M. L. (1991). A compendium of textbook views on planned versus post hoc tests. In B. Thompson (Ed.), Advances in education research: Substantive findings, methodological developments (Vol. 1, pp. 107-118). Greenwich, CT: JAI Press.
77. Tukey, J. W. (1953). The problem of multiple comparisons. In H. Braun (Ed.), The collected works of John W. Tukey, Vol VIII: Multiple comparisons: 1948-1983 (pp. 1-300). New York, NY: Chapman & Hall.
78. U.S. Department of Health and Human Services. (1998). Guidance for industry: E9 statistical principles for clinical trials. Retrieved from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm073137.pdf
79. Wang, L. (1993, November). Planned versus unplanned contrasts: Exactly why planned contrasts tend to have more power against Type II error. Paper presented at the Annual Meeting of the Mid-South Educational Research Association, New Orleans, LA. Retrieved from http://files.eric.ed.gov/fulltext/ED364598.pdf
80. Weiss, D. J. (2006). Analysis of variance and functional measurement. New York, NY: Oxford University Press.
81. Weisse, K., Winkler, S., Hirche, F., Herberth, G., Hinz, D., Bauer, . . . Lehmann, I. (2013). Maternal and newborn vitamin D status and its impact on food allergy development in the German LINA cohort study. Allergy, 68(2), 220-228.
82. Welkowitz, J., Cohen, B. H., & Lea, R. B. (2012). Introductory statistics for the behavioral sciences (7th ed.). Hoboken, NJ: John Wiley.
83. Zieffler, A. S., Harring, J. R., & Long, J. D. (2011). Comparing groups: Randomization and bootstrap methods using R. Hoboken, NJ: John Wiley.