Lopsided reasoning on lopsided tests and multiple comparisons

Australian and New Zealand Journal of Statistics

Volumn 54, Issue 1, 2012, Pages 23-42

  Hurlbert, Stuart H ^a   Lombardi, Celia M ^b  

^a SAN DIEGO STATE UNIVERSITY   (United States)

^b MUSEO ARGENTINO DE CIENCIAS NATURALES BERNARDINO RIVADAVIA   (Argentina)

Author keywords

Bonferroni procedure; Comparison wise error rate; Directed tests; False discovery rate; Family wise error rate; Multiplicity; One tailed tests; Randomized clinical trials; Set wise error rate; Significance tests; Split tailed tests; Type I error

Indexed keywords

EID: 84868588306     PISSN: 13691473     EISSN: 1467842X     Source Type: Journal    
DOI: 10.1111/j.1467-842X.2012.00652.x     Document Type: Review

References (141)

1. ABELSON, R.P. (1995). Statistics as Principled Argument . Hillsdale, NJ: Lawrence Erlbaum.
2. ALTMAN, D.G. (1991). Practical Statistics for Medical Research . New York: Chapman and Hall.
3. BART, J., FLIGNER, M.A. & NOTZ, W.I. (1998). Sampling and Statistical Methods for Behavioral Ecologists. Cambridge: Cambridge University Press.
4. BENDER, R. & LANGE, S. (2001). Adjusting for multiple testing - when and how? J. Clin. Epidemiol. 54, 343-349.
5. BENJAMINI, Y.&HOCHBERG,Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Stat. Soc. Ser. B Stat. Methodol. 57, 289-300.
6. BENJAMINI, Y. & YEKUTIELI, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals Statist. 29, 1165-1188.
7. BENJAMINI, Y. & YEKUTIELI, D. (2005). Quantitative trait loci analysis using the false discovery rate. Genetics 171, 783-790.
8. BENJAMINI, Y., DRAI, D., ELMER, G., KAFKAFI, N. & GOLANI, I. (2001). Controlling the false discovery rate in behavior genetics research. Behav. Brain Res. 125, 279-284.
9. BENJAMINI, Y., KRIEGER, A.M. & YEKUTIELI, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93, 491-507.
10. BOX, G.E.P., HUNTER, W.G. & HUNTER, J.S. (2005). Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building, 2nd edn. New York: Wiley.
11. BRAUN, H.I., ed. (1994). The Collected Works of John W. Tukey, Volume VIII, Multiple Comparisons: 1948-1983. New York: Chapman and Hall.
12. BRAVER, S.L. (1975). On splitting the tails unequally: a new perspective on one- versus two-tailed tests. Educ. Psychol. Meas. 35, 283-301.
13. CARMER, S.G. & WALKER, W.M. (1982). Baby Bear's dilemma: A statistical tale. Agron. J. 74, 122-124.
14. CHRISTENSEN, R. (2005). Testing Fisher, Neyman, Pearson, and Bayes. Amer. Statist. 59, 121-126.
15. COWLES, M. (1989). Statistics in Psychology: An Historical Perspective. Hillsdale, NJ: Lawrence Erlbaum.
16. COX, D.R. (1958). Some problems connected with statistical inference. Ann. Math. Statist. 29, 357-372.
17. COX, D.R. (1965). A remark on multiple comparison methods. Technometrics 7, 223-224.
18. COX, D.R. (2006). Principles of Statistical Inference. Cambridge: Cambridge University Press.
19. CRABBE, J.C.,WAHLSTEN,D.&DUDEK,B.C. (1999).Genetics ofmouse behavior: interactions with laboratory environment. Science 284, 1670-1672.
20. CUI, X., HWANG, J.T., QIU, J., BLADES, N.J. & CHURCHILL, G.A. (2005). Improved statistical tests for differential gene expression by shrinking variance components estimates. Biostatistics 8, 414-432.
21. CURRAN-EVERETT, D. (2000). Multiple comparisons: Philosophies and illustrations. Amer. J. Physiol.: Regul., Integr. Comp. Physiol. 279, R1-R8.
22. DARLINGTON, R.B.&CARLSON, P.M. (1987). Behavioral Statistics. New York: The Free Press.
23. DAY, R.W. & QUINN, G.P. (1989). Comparison of treatments after an analysis of variance in ecology. Ecol. Monogr. 59, 433-463.
24. DMITRIENKO, A. & TAMHANE, A.C. (2007). Gatekeeping procedures with clinical trial applications. Pharm. Stat. 6, 171-180.
25. DMITRIENKO, A., OFFEN, W.W. & WESTFALL, P.H. (2003). Gatekeeping strategies for clinical trials that do not require all primary effects to be significant. Stat. Med. 22, 2387-2400.
26. DMITRIENKO, A., MOLENBERGHS, G., CHUANG-STEIN, C. & OFFEN, W. (2005). Analysis of Clinical Trials Using SAS: A Practical Guide. Cary, NC: SAS Institute.
27. DMITRIENKO, A., WIENS, B.L., TAMHANE, A.C. & WANG, X. (2007). Tree-structured gatekeeping tests in clinical trials with hierarchically ordered objectives. Stat. Med. 26, 2465-2478.
28. DUDOIT, D., SHAFFER, J.P. & BOLDRICK, J.C. (2003). Multiple hypothesis testing in microarray experiments. Statist. Sci. 18, 71-103.
29. DUNCAN, D.B. (1965). A Bayesian approach to multiple comparisons. Technometrics 7, 171-222.
30. ECKLUND,G.&SEEGER, P. (1965).Massignifikansanalys. Statistisk Tidskrift Stockholm, 3d series 4, 355-365.
31. EYSENCK, H.J. (1960). The concept of statistical significance and the controversy about one-tailed tests. Psych. Rev. 67, 269-271.
32. FARCOMENI, A. (2008). A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Stat. Methods Med. Res. 17, 347-388.
33. FERNANDO, R.L., NETTLETON, D., SOUTHEY, R.R., DEKKERS, J.C.M., ROTHSCHILD, M.F. & SOLLER, M. (2004). Controlling the proportion of false positives in multiple dependent tests. Genetics 166, 611- 619.
34. FINNEY, D.J. (1988). Was this in your statistics textbook? III. Design and analysis. Exp. Agric. 24, 421-432.
35. FLEISS, J.L. (1981). Statistical Methods for Rates and Proportions, 2nd edn. New York: Wiley.
36. FLEISS, J.L. (1986). The Design and Analysis of Clinical Experiments. New York: Wiley.
37. FREEDMAN, D., PISANI, R., PURVES, R. & ADHIBARI, A. (1991). Statistics, 2nd edn. New York: Norton.
38. GAINES, S.D. & RICE, W.R. (1990). Analysis of biological data when there are ordered expectations. Amer. Nat. 135, 310-317.
39. GARCIA, L.V. (2004). Escaping the Bonferroni iron claw in ecological studies. Oikos 105, 657-663.
40. GENOVESE, C.R., LAZAR, N.A. & NICHOLS, T. (2002). Threshholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15, 870-878.
41. GENOVESE, C.R., ROEDER, K. & WASSERMAN, L. (2006). False discovery control with p-value weighting. Biometrika 93, 509-524.
42. GLASS G.V. & HOPKINS, K.D. (1984). Statistical Methods in Education and Psychology, 2nd edn. Englewood Cliffs, NJ: Prentice-Hall.
43. GOLDFRIED, M.R. (1959). One-tailed tests and 'unexpected' results. Psych. Rev. 66, 79-80.
44. HARNETT, D.L. & SONI, A.K. (1991). Statistical Methods for Business and Economics, 4th edn. New York: Addison-Wesley.
45. HARRIS R.J. (2005a). Classical statistical inference extended: split-tailed tests. Encyclopedia of Statistics in Behavioral Science, vol. 1, eds. B.S. Everitt and D.C. Howell. pp. 263-268. Chichester: Wiley.
46. HARRIS R.J. (2005b). Classical statistical inference: practice versus presentation. Encyclopedia of Statistics in Behavioral Science, vol. 1, eds. B.S. Everitt and D.C. Howell. pp. 268-278. Chichester: Wiley.
47. HARRIS, R.J. & QUADE, D. (1992). The minimally important difference significant criterion for sample size. J. Educ. Behav. Stat. 17, 27-49.
48. HAWKINS, D. (2005). Biomeasurement . New York: Oxford University Press.
49. HOCHBERG, Y. & TAMHANE, A.C. (1987). Multiple Comparison Procedures. New York: Wiley.
50. HOOVER, D.K. & SIEGLER, M.V. (2008). Sound and fury: McCloskey and significance testing in economics. J. Econ. Methodol. 15, 1-37.
51. HUNG, H.M.J.&WANG, S.-J. (2009). Some controversialmultiple testing problems in regulatory applications. J. Biopharm. Statist. 19, 1-11.
52. HURLBERT, S.H. (1990). Pastor binocularis: now we have no excuse [Review of The Design of Experiments, by R. Mead]. Ecology 71, 1222-1223.
53. HURLBERT, S.H. & LOMBARDI, C.M. (2003). Design and analysis: uncertain intent, uncertain result [Review of Experimental Design and Data Analysis for Biologists, by G.P. Quinn & M.J. Keough]. Ecology 83, 810-812.
54. HURLBERT, S.H. & LOMBARDI, C.M. (2009). Final collapse of the Neyman-Pearson decision-theoretic framework and rise of the neoFisherian. Ann. Zool. Fenn. 46, 311-349.
55. ICH (1999). ICH harmonised tripartite guideline: statistical principles for clinical trials. E9 Expert Working Group, International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use. Stat. Med. 18, 1905-1942.
56. KAISER, H.F. (1960). Directional statistical decisions. Psych. Rev. 67, 160-167.
57. KENDALL, M. & STUART, A. (1979). The Advanced Theory of Statistics, 2, Inference and Relationship, 4th edn. London: Griffin.
58. KEPPEL, G. (1991). Design and Analysis: A Researcher's Handbook , 3rd edn. Englewood Cliffs, NJ: Prentice Hall.
59. KIMMEL, H.D. (1957). Three criteria for the use of one-tailed tests. Psych. Bull. 16, 345-353.
60. KIRK, R.E. (1982). Experimental Design, 2nd edn. Pacific Grove, CA: Brooks/Cole Publishing.
61. KLINE, R.B. (2004). Beyond Significance Testing. Washington, DC: American Psychological Association.
62. KORNILOV, S.G. (1993). Errors in the description of the F test and some thoughts on one-sided statistical tests. Industr. Lab. 59, 720-725. (Translation of Russian article published in: Zavodskaya Laboratoriya 59, 60, 1993).
63. LEEK, J.T. & STOREY, J.D. (2008). A general framework for multiple testing dependence. Proc. Natl. Acad. Sci. USA 105, 18718-18723.
64. LITTLE, T.M. (1978). If Galileo published in HortScience. HortScience 13, 504-506.
65. LOMBARDI, C.M. & HURLBERT, S.H. (2009).Misprescription and misuse of one-tailed tests. Austral Ecol. 34, 447-468.
66. MANTEL, N. (1983). Ordered alternatives and the 1-1/2 tail test. Amer. Statist. 37, 225-228.
67. MARCUS, R., PERITZ, E. & GABRIEL, K.R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63, 655-660.
68. MARTIN, P. & BATESON, P. (2007). Measuring Behaviour: An Introductory Guide, 3rd edn. Cambridge: Cambridge University Press.
69. MEAD, R. (1988). The Design of Experiments. Cambridge: Cambridge University Press.
70. MEAD R. & CURNOW, R.N. (1983). Statistical Methods in Agriculture and Experimental Biology. NewYork: Chapman and Hall.
71. MEAD, R., CURNOW, R.N. & HASTED, A.M. (2003). Statistical Methods in Agriculture and Experimental Biology, 3rd edn. New York: Chapman & Hall.
72. MEEK, G.E. & OZGUR, C.O. (2004). Unequal division of type I risk in statistical inferences. Decision Sci. J. Innov. Educ. 2, 45-55.
73. MEEK, G.E. & TURNER, S.J. (1983). Statistical Analysis for Business Decisions. Upper Saddle River, NJ: Houghton and Mifflin.
74. MILLER, R.G., JR. (1981). Simultaneous Statistical Inference, 2nd edn. New York: Springer.
75. MOORE, D.S. & MCCABE, G.P. (1989). Introduction to the Practice of Statistics. New York: W.H. Freeman.
76. MORAN, M.D. (2003). Arguments for rejecting the sequential Bonferroni in ecological studies. Oikos 100, 403-405.
77. MOYé, L.A. (2000). Statistical Reasoning in Medicine: The Intuitive P-value Primer . New York: Springer.
78. MOYé, L.A. (2003). Multiple Analyses in Clinical Trials: Fundamentals for Investigators. New York: Springer.
79. MOYé, L.A. (2006a). Statistical Monitoring of Clinical Trials. New York: Springer.
80. MOYé, L.A. (2006b). Statistical Reasoning in Medicine: The Intuitive P-value Primer , 2nd edn. New York: Springer.
81. MOYé, L.A. (2008). Disciplined analyses in clinical trials: The dark heart of the matter. Stat.Meth. Med. Res. 17, 253-264.
82. NAKAGAWA, S. (2004). A farewell to Bonferroni: the problems of low statistical power and publication bias. Behav. Ecol. 15, 1044-1045.
83. NEALE, M.C. & MILLER, M.B. (1996). The use of likelihood-based confidence intervals in genetic models. Behav. Genet. 27, 113-120.
84. NEUHAUS, K.-L., VON ESSEN, R., TEBBE, U., VOGT, A., ROTH, M., REISS, M., NIEDERER, W., FORYCKI, F., WIRTZFELD,A.,MAEURER,W., LIMBOURG, P.,MERX,W.&HAERTEN,K. (1992). Improved thrombolysis in acute myocardial infarctionwith front-loaded administration ofAlteplase: results of the rt-PA-APSAC patency study (TAPS). J. Amer. Coll. Cardiol. 19, 885-891.
85. NICKERSON, R.S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psych. Meth. 5, 241-301.
86. NOSANCHUK, T.A. (1978). Serendipity tails: a note on two tailed hypothesis tests with asymmetric regions of rejection. Acta Sociol., 21, 249-253.
87. OAKES, M. (1986). Statistical Inference: A Commentary for the Social and Behavioural Sciences. New York: Wiley.
88. O'BRIEN, P.C. (1983). The appropriateness of analysis of variance and multiple-comparison procedures. Biometrics 93, 787-788.
89. O'KEEFE, D.J. (2003). Colloquy: Should familywise alpha be adjusted? Against familywise alpha adjustment. Human Commun. Res. 29, 431-447.
90. O'NEILL, R.&WETHERILL, G.B. (1971). The present state of multiple comparison methods. J. Roy. Stat. Soc. Ser. B Stat. Methodol. 33, 218-250.
91. PEARCE, S.C. (1993). Data analysis in agricultural experimentation. III. Multiple comparisons. Exper. Agric. 29, 1-8.
92. PERNEGER, T.V. (1998). What's wrong with Bonferroni adjustments. Brit. Med. J. 316, 1236-1238.
93. PERRY, J.N. (1986). Multiple-comparison procedures: a dissenting view. J. Econ. Entomol. 79, 1149- 1155.
94. PILLEMER, D.B. (1991). One- versus two-tailed hypothesis tests in contemporary educational research. Educ. Researcher 20(9), 13-17.
95. POCOCK, S.J. (1997). Clinical trials with multiple outcomes: a statistical perspective on their design, analysis, and interpretation. Controll. Clin. Trials 18, 530-545.
96. PREECE, D.A. (1982). The design and analysis of experiments: What has gone wrong? Util. Math. 21A, 210-244.
97. QUINN, G.P. & KEOUGH, M.J. (2002). Experimental Design and Data Analysis for Biologists. Cambridge: Cambridge University Press.
98. RAMSEY, P.H. (1990). 'One-and-a-half-tailed' tests of significance. Psych. Reports 66, 653-654.
99. RICE, W.R. & GAINES, S.D. (1994a). Extending nondirectional heterogeneity tests to evaluate simple ordered alternative hypotheses. Proc. Natl. Acad. Sci. USA 91, 225-226.
100. RICE, W.R. & GAINES, S.D. (1994b). The ordered heterogeneity family of tests. Biometrics 50, 746-752.
101. RICE, W.R. & GAINES, S.D. (1994c). 'Heads I win, tails you lose': Testing directional alternative hypotheses in ecological and evolutionary research. Trends Ecol. Evol. 9, 235-237.
102. ROTHMAN, K.J. (1990). No adjustments are needed for multiple comparisons. Epidemiology 1, 43-46.
103. SAVILLE, D.J. (1990). Multiple comparison procedures: the practical solution. Amer. Statist. 44, 174-180.
104. SAVITZ, D.A. & OLSHAN, A.F. (1995). Multiple comparisons and related issues in the interpretation of epidemiologic data. Am. J. Epidemiol. 142, 904-908.
105. SAVITZ, D.A. & OLSHAN, A.F. (1998). Describing data requires no adjustment for multiple comparisons: a reply from Savitz and Olshan. Am. J. Epidemiol. 147, 813.
106. SCHULMAN, R.S. (1992). Statistics in Plain English . New York: Van Nostrand Reinhold.
107. SCHULZ, K.F. & GRIMES, D.A. (2005). Multiplicity in randomized trials I: Endpoints and treatments. Lancet 365, 1591-1595.
108. SEEGER, P. (1968). A note on method for the analysis of significances en masse. Technometrics 10, 586-593.
109. SENN, S. & BRETZ, F. (2007). Power and sample size when multiple endpoints are considered. Pharm. Statist. 6, 161-170.
110. SHAFFER, J.P. (1972). Directional statistical hypotheses and comparisons among means. Psych. Bull. 77, 195-197.
111. SHAFFER, J.P. (1995). Multiple hypothesis testing. Ann. Rev. Psychol. 46, 561-584.
112. SHAFFER, J.P. (2006). Recent developments towards optimality in multiple hypothesis testing. IMS Lecture Notes-Monograph Series 49, 16-32.
113. SIEGEL, S.,(1956). Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill.
114. SIEGEL, S. & CASTELLAN, N.J., Jr. (1988) Nonparametric Statistics for the Behavioral Sciences, 2d edn. New York: McGraw-Hill.
115. SINGH, A. & DAN, I. (2006). Exploring the false discovery rate in multichannel NIRS. Neuroimage 33, 542-549.
116. SKIPPER, K.S., Jr., GUENTHER, A.L. & NASS, G. (1967). The sacredness of .05: a note concerning the uses of statistical levels of significance in social science. Amer. Sociol. 2, 16-18.
117. SNEDECOR, G.W. & COCHRAN, W.G. (1989). Statistical Methods, 8th edn. Ames, IO: Iowa State University Press.
118. SOKAL, R.R. & ROHLF, F.J. (1995). Biometry, 3rd edn. San Francisco: Freeman.
119. SORIC, B. (1989). Statistical "discoveries" and effect-size estimation. J. Amer. Statist. Assoc. 84, 608-610.
120. SOTO, D. & HURLBERT, S.H. (1991). Long-term experiments on calanoid-cyclopoid interactions. Ecol. Monogr. 61, 245-265.
121. SPANOS, A. (1999). Probability Theory and Statistical Inference: Econometric Modeling with Observational Data. Cambridge: Cambridge University Press.
122. SPANOS, A. (2008). Review of The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice and Lives, by S.T. Ziliak and D.N. McCloskey. Erasmus J. Philos. Econ. 1, 154-164.
123. SPJOTVOLL, E. (1972). On the optimality of some multiple comparison procedures. Ann. Math. Statist. 43, 398-411.
124. STEWART-OATEN, A. (1995). Rules and judgments in statistics: three examples. Ecology 76, 2001-2009.
125. STOREY, J.D. (2003). The positive false discovery rate: A Bayesian interpretation and the q-value. Ann. Statist. 6, 2013-2035.
126. STOREY, J.D. (2007). The optimal discovery procedure: a new approach to simultaneous significance testing. J. R. Stat. Soc. Ser. B Stat. Methodol. 69, 347-368.
127. STOREY, J.D., TAYLOR, J.E. & SIEGMUND, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. J. R. Stat. Soc. Ser. B Stat. Methodol. 66, 187-205.
128. STOREY, J.D., DAI, J.Y. & LEEK, J.T. (2007). The optimal discovery procedure for large-scale significance testing, with applications to comparative microarray experiments. Biostatistics 8, 414-432.
129. TUKEY, J.W. (1953). The problem of multiple comparisons. The Collected Works of John W. Tukey, Volume III, ed. H.I. Braun, pp. 1-300 [1994]. New York: Chapman and Hall. [Work completed and privately circulated starting in 1953].
130. TUKEY, J.W. (1977). Some thoughts on clinical trials, especially problems of multiplicity. Science, 198, 679-684.
131. TUKEY, J.W. (1991). The philosophy of multiple comparisons. Stat. Sci. 6, 100-116.
132. UNDERWOOD, A.J. (1997). Experiments in Ecology. London: Blackwell.
133. VERHOEVEN, K.J.F. (2005). Implementing false discovery rate control: increasing your power. Oikos 108, 643-647.
134. WELKOWITZ, J., EWEN, R.B. & COHEN, J. (1971, 1991, 1999). Introductory Statistics for the Behavioral Sciences, 1st, 4th, 5th edns. New York: Harcourt Brace Jovanovich.
135. WELKOWITZ, J., COHEN, B.H. & EWEN, R.B. (2006). Introductory Statistics for the Behavioral Sciences, 6th edn. New York: Wiley.
136. WESTFALL, P.H. & KRISHEN, A. (2001). Optimally weighted, fixed sequence and gatekeeper multiple testing procedures. J. Statist. Plann. Inference 99, 25-40.
137. WESTFALL, P.H. & YOUNG, S.S. (1993). Resampling-Based Multiple Testing: Examples and Methods for p-value Adjustment. New York: Wiley.
138. WILSON, W. (1962). A note on the inconsistency inherent in the necessity to perform multiple comparisons. Psych. Bull. 59, 296-300.
139. YATES, R. & HEALY,M.J.R. (1964). How should we reform the teaching of statistics? J. Roy. Statist. Soc. Ser. A 127, 199-210.
140. ZAR, J.H. (2004) Biostatistical Analysis, 4th edn. New York: Prentice-Hall, Inc.
141. ZILIAK, S.T. & MCCLOSKEY, D.N. (2008). The Cult of Statistical Significance: How the Standard Error Cost Us Jobs, Justice and Lives. Ann Arbor, MI: University of Michigan Press.