Researchers should make thoughtful assessments instead of null-hypothesis significance tests

Organization Science

Volumn 22, Issue 4, 2011, Pages 1105-1120

  Schwab, Andreas ^a   Abrahamson, Eric ^b   Starbuck, William H ^c   Fidler, Fiona ^d  

^a IOWA STATE UNIVERSITY   (United States)

^b The Sanford C Bernstein Co Center for Leadership and Ethics   (United States)

^c UNIVERSITY OF OREGON   (United States)

^d LA TROBE UNIVERSITY   (Australia)

Author keywords

Analyses; Research design and methods; Statistics

Indexed keywords

EID: 79960850779     PISSN: 10477039     EISSN: 15265455     Source Type: Journal    
DOI: 10.1287/orsc.1100.0557     Document Type: Article

References (94)

1. Aguinis, H., S. Werner, J. L. Abbott, C. Angert, J. H. S. Park, D. Kohlhausen. 2010. Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organ. Res. Methods 13(3) 515-539.
2. Algina, J., H. J. Keselman. 2003. Approximate confidence intervals for effect sizes. Ed. Psych. Measurement 63(4) 537-553.
3. American Educational Research Association. 2006. Standards for reporting on empirical social science research in AERA publications. Ed. Res. 35(6) 33-40.
4. American Psychological Association. 2010. Publication Manual of the American Psychological Association. American Psychological Association, Washington, DC.
5. Armstrong, J. S. 2007. Significance tests harm progress in forecasting. Internat. J. Forecasting 23(2) 321-327.
6. Berkson, J. 1938. Some difficulties of interpretation encountered in the application of the chi-square test. J. Amer. Statist. Assoc. 33(203) 526-536.
7. Bowman, E. H. 1984. Content analysis of annual reports for corporate strategy and risk. Interfaces 14(1) 61-71.
8. Box, G. E. P., N. R. Draper. 1969. Evolutionary Operation. Wiley, New York.
9. Breaugh, J. A. 2003. Effect size estimation: Factors to consider and mistakes to avoid. J. Management 29(1) 79-97.
10. Cadman, L., A. Findlay. 1998. Assessing practice nurses' change in nutrition knowledge following training from a primary care dietitian. J. Royal Soc. Promotion Health 118(4) 206-209.
11. Capraro, R. M., M. Capraro. 2002. Treatments of effect sizes and statistical significance tests in textbooks. Ed. Psych. Measurement 62(5) 771-782.
12. Claudy, J. G. 1972. Comparison of five variable weighting procedures. Ed. Psych. Measurement 32(2) 311-322.
13. Cleveland, W. S. 1985. The Elements of Graphing Data. Wadsworth, Monterey, CA.
14. Cohen, J. 1994. The earth is round (p < 0005). Amer. Psych. 49(12) 997-1003.
15. Colhoun, H. M., P. M. McKeigue, G. Davey Smith. 2003. Problems of reporting genetic associations with complex outcomes. Lancet 36(9360) 865-872.
16. Connor, E. F., D. Simberloff. 1986. Competition, scientific method, and null models in ecology. Amer. Sci. 74(2) 155-162.
17. Cortina, J. M., R. G. Folger. 1998. When is it acceptable to accept a null hypothesis: No way, Jose? Organ. Res. Methods 1(3) 334-350.
18. Cortina, J. M., H. Nouri. 1999. Effect Size for ANOVA Designs. Sage, Beverly Hills, CA.
19. Cumming, G., S. Finch. 2001. A primer on the understanding, use and calculation of confidence intervals that are based on central and noncentral distributions. Ed. Psych. Measurement 61(4) 532-574.
20. Darlington, M. L. 1973. Comparing two groups by simple graphs. Psych. Bull. 79(2) 110-116.
21. Doksum, K. A. 1977. Some graphical methods in statistics: A review and some extensions. Statistica Neerlandica 31(2) 53-68.
22. Dorans, N., F. Drasgow. 1978. Alternative weighting schemes for linear prediction. Organ. Behav. Human Perform. 21(3) 316-345.
23. Einhorn, H. J., R. M. Hogarth. 1975. Unit weighting schemes for decision making. Organ. Behav. Human. Perform. 13(2) 171-192.
24. Elliott, J. W. 1973. A direct comparison of short-run GNP forecasting models. J. Bus. 46(1) 33-60.
25. Ellis, P. 2010. The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis and the Interpretation of Research Results. Cambridge University Press, Cambridge, UK.
26. Faust, D. 1984. The Limits of Scientific Reasoning. University of Minnesota Press, Minneapolis.
27. Fidler, F. 2005. From statistical significance to effect estimation: Statistical reform in psychology, medicine and ecology. Doctoral dissertation, University of Melbourne, Melbourne, VIC, Australia.
28. Fidler, F., G. Cumming, M. Burgman, N. Thomason. 2004. Statistical reform in medicine, psychology and ecology. J. Socio-Econom. 33 615-630.
29. Fidler, F., G. Cumming, N. Thomason, D. Pannuzzo, J. Smith, P. Fyffe, H. Edmonds, C. Harrington, R. Schmitt. 2005. Evaluating the effectiveness of editorial policy to improve statistical practice: The case of the journal of consulting and clinical psychology. J. Consulting Clinical Psych. 73(1) 136-143.
30. Fiol, C. M. 1989. A semiotic analysis of corporate language: Organizational boundaries and joint venturing. Admin. Sci. Quart. 34(2) 277-303.
31. Fisher, R. A. 1925. Statistical Methods for Research Workers. Oliver and Boyd, Edinburgh, UK.
32. Fleiss, J. L. 1986. Significance tests do have a role in epidemiological research: Reactions to A. A. Walker. Amer. J. Public Health 76(5) 559-560.
33. Gauch, H. G. 2002. Scientific Method in Practice. Cambridge University Press, Cambridge, UK.
34. Gauch, H. G. 2006. Winning the accuracy game. Amer. Sci. 94(2) 135-143.
35. Goldberg, L. R. 1970. Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inference. Psych. Bull. 73(6) 422-432.
36. Greenwald, A. G. 1975. Consequences of prejudice against the null hypothesis. Psych. Bull. 82(1) 1-20.
37. Greiser, C. M., E. M. Greiser, M. Dören. 2005. Menopausal hormone therapy and risk of breast cancer: A meta-analysis of epidemiological studies and randomized controlled trials. Human Reproduction Update 11(6) 561-573.
38. Grissom, R. J., J. J. Kim. 2005. Effect Sizes for Research: A Broad Practical Approach. Lawrence Erlbaum Associates, Mahwah, NJ.
39. Haller, H., S. Krauss. 2002. Misinterpretations of significance: A problem students share with their teachers? Methods Psych. Res. 7(1) 1-20.
40. Hubbard, R., J. S. Armstrong. 1992. Are null results becoming an endangered species in marketing? Marketing Lett. 3(2) 127-136.
41. Hubbard, R., J. S. Armstrong. 2006. Why we don't really know what statistical significance means: Implications for educators. J. Marketing Ed. 28(2) 114-120.
42. Hubbard, R., M. J. Bayarri. 2003. Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing. Amer. Statistician 57(3) 171-178.
43. Ioannidis, J. P. A. 2003. Genetic associations: False or true? Trends Molecular Med. 9(4) 135-138.
44. Ioannidis, J. P. A. 2005a. Contradicted and initially stronger effects in highly cited clinical research. J. Amer. Medical Assoc. 294(2) 218-228.
45. Ioannidis, J. P. A. 2005b. Why most published research findings are false. PLoS Med. 2(8) e124.
46. Jeffreys, W. H., J. O. Berger. 1992. Ockham's razor and Bayesian analysis. Amer. Sci. 80(1) 64-72.
47. John, I. D. 1992. Statistics as rhetoric in psychology. Australian Psych. 27(3) 144-149.
48. Kendall, P. C. 1997. Editorial. J. Consulting Clinical Psych. 65(1) 3-5.
49. Keselman, H. J., R. R. Wilcox, L. M. Lix, J. Algina, K. Fradette. 2007. Adaptive robust estimation and testing. British J. Math. Statist. Psych. 60(2) 267-293.
50. Kline, R. B. 2004. What's wrong with statistical tests-And where we go from here. R. Kline, ed. Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research, Chapter 3. APA Books, Washington, DC, 61-91.
51. Langman, M. J. S. 1986. Towards estimation and confidence intervals. British Med. J. 292(6522) 716.
52. Levinthal, D. A. 1991. Random walks and organizational mortality. Admin. Sci. Quart. 36(3) 397-420.
53. Lykken, D. T. 1968. Statistical significance in psychological research. Psych. Bull. 70(3, Part 1) 151-159.
54. Makridakis, S., A. Andersen, R. Carbone, R. Fildes, M. Hibon, R. Lewandowski, J. Newton, E. Parzen, R. L. Winkler. 1982. The accuracy of extrapolation (time series) methods: Results of a forecasting competition. J. Forecasting 1 111-153.
55. Mayo, D. 2006. Philosophy of statistics. S. Sarkar, J. Pfeifer, eds. The Philosophy of Science: An Encyclopedia. Routledge, London, 802-815.
56. Meehl, P. E. 1954. Clinical versus Statistical Prediction: A Theoretical Analysis and Review of the Evidence. University of Minnesota Press, Minneapolis.
57. Meehl, P. E. 2002. Interview by Fiona Fidler. August 27. University of Minnesota, Minneapolis.
58. Mezias, J. M., W. H. Starbuck. 2003. Studying the accuracy of managers' perceptions: A research odyssey. British J. Management 14(1) 3-17.
59. Oakes, M. 1986. Statistical Inference. A Commentary for the Social and Behavioural Sciences. Wiley, Chichester, UK.
60. Pant, P. N., W. H. Starbuck. 1990. Innocents in the forest: Forecasting and research methods. J. Management 16(2) 433-460.
61. Popper, K. R. 1959. The Logic of Scientific Discovery. Basic Books, New York.
62. Powell, T. C. 2003. Varieties of competitive parity. Strategic Management J. 24(1) 61-86.
63. Rosenberg, B., M. Houglet. 1974. Error rates in CRSP and Compustat data bases and their implications. J. Finance 29(4) 1303-1310.
64. Rosnow, R., R. Rosenthal. 1989. Statistical procedures and the justification of knowledge in psychological science. Amer. Psych. 44(10) 1276-1284.
65. Rousseau, D. M., J. Manning, D. Denyer. 2008. Evidence in management and organizational science: Assembling the field's full weight of scientific knowledge through syntheses. Acad. Management Ann. 2(1) 475-515.
66. Rousseeuw, P. J., A. M. Leroy. 1987. Robust Regression and Outlier Detection. Wiley, New York.
67. Rothman, K. J. 1998. Writing for epidemiology. Epidemiology 9(3) 333-337.
68. Salancik, G. R., J. R. Meindl. 1984. Corporate attributions as strategic illusions of management control. Admin. Sci. Quart. 29(12) 238-254.
69. San Miguel, J. G. 1977. The reliability of R&D data in Compustat and 10-K Reports. Accounting Rev. 52(3) 638-641.
70. Schmidt, F. L. 1971. The relative efficiency of regression and simple unit predictor weights in applied differential psychology. Ed. Psych. Measurement 31(3) 699-714.
71. Schmidt, F. L. 1996. Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psych. Methods 1(2) 115-129.
72. Schmidt, F. L., J. E. Hunter. 1997. Eight common but false objections to the discontinuation of significance testing in analysis of research data. L. Harlow, S. Mulaik, J. Steiger, eds. What If There Were No Significance Tests? Lawrence Erlbaum Associates, Mahwah, NJ, 37-63.
73. Schwab, A., W. H. Starbuck. 2009. Null-hypothesis significance tests in behavioral and management research: We can do better. D. Bergh, D. Ketchen, eds. Research Methodology in Strategy and Management, Vol. 5. Elsevier, New York, 29-54.
74. Seth, A., K. D. Carlson, D. E. Hatfield, H. W. Lan. 2009. So what? Beyond statistical significance to substantive significance in strategy research. D. Bergh, D. Ketchen, eds. Research Methodology in Strategy and Management, Vol. 5. Elsevier, New York, 3-28.
75. Shah, N. R., J. Borenstein, R. W. Dubois. 2005. Postmenopausal hormone therapy and breast cancer: A systematic review and metaanalysis. Menopause 12(6) 668-678.
76. Shrout, P. E. 1997. Should significance tests be banned? Psych. Sci. 8(1) 1-2.
77. Smithson, M. 2001. Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Ed. Psych. Measurement 61(4) 605-632.
78. Soofi, E. S., P. C. Nystrom, M. Yasai-Ardekani. 2009. Executives' perceived environmental uncertainty shortly after 9/11. Comput. Statist. Data Anal. 53(9) 3502-3515.
79. Starbuck, W. H. 1994. On behalf of naïveté. J. A. C. Baum, J. V. Singh, eds. Evolutionary Dynamics of Organizations. Oxford University Press, New York, 205-220.
80. Starbuck, W. H. 2006. The Production of Knowledge: The Challenge of Social Science Research. Oxford University Press, Oxford, UK.
81. Steiger, J. H., R. T. Fouladi. 1992. R2-A computer-program for interval estimation, power calculations, sample-size estimation, and hypothesis testing in multiple regression. Behav. Res. Methods, Instruments, Comput. 24(4) 581-582.
82. Task Force on Statistical Significance. 1996. Initial report. Board of Scientific Affairs, American Psychological Association, Washington, DC.
83. Thompson, B. 1995. Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Ed. Psych. Measurement 55(4) 525-534.
84. Thompson, B. 1999a. Why "encouraging" effect size reporting is not working: The etiology of researcher resistance to changing practices. J. Psych. 133(2) 133-140.
85. Thompson, B. 1999b. Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Ed. Psych. Rev. 11(2) 157-169.
86. Thompson, B. 2002. What future quantitative social science research could look like: Confidence intervals for effect sizes. Ed. Res. 31(3) 25-32.
87. Thompson, B. 2006. Foundations of Behavioral Statistics: An Insight-Based Approach. Guilford, New York.
88. Tukey, J. W. 1991. The philosophy of multiple comparisons. Statist. Sci. 6(1) 100-116.
89. Vacha-Haase, T., J. E. Nilsson, D. R. Reetz, T. S. Lance, B. Thompson. 2000. Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory Psych. 10(3) 413-425.
90. Wacholder, S., S. Chanock, M. Garcia-Closas, L. El ghormli, N. Rothman. 2004. Assessing the probability that a positive report is false: An approach for molecular epidemiology studies. J. Natl. Cancer Inst. 96(6) 434-442.
91. Webster, E. J., W. H. Starbuck. 1988. Theory building in industrial and organizational psychology. C. L. Cooper, I. Robertson, eds. International Review of Industrial and Organizational Psychology 1988. Wiley, London, 93-138.
92. Wilcox, R. R. 1997. Introduction to Robust Estimation and Hypothesis Testing. Academic Press, San Diego.
93. Wilk, M. B., R. Gnanades. 1968. Probability plotting methods for analysis of data. Biometrika 55(1) 1-17.
94. Wilkinson, L. 1999. APA Task Force on Statistical Inference. Statistical methods in psychology journals: Guidelines and explanations. Amer. Psych. 54(8) 594-604.