Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind

Organizational Research Methods

Volumn 13, Issue 3, 2010, Pages 515-539

  Aguinis, Herman ^a   Werner, Steve ^b   Abbott, JéAnna Lanza ^b   Angert, Cory ^b   Park, Hyung Joon ^b   Kohlhausen, Donna ^b  

^a INDIANA UNIVERSITY   (United States)

^b UNIVERSITY OF HOUSTON   (United States)

Author keywords

hypothesis testing; practical significance; statistical significance

Indexed keywords

EID: 77956479035     PISSN: 10944281     EISSN: 15527425     Source Type: Journal    
DOI: 10.1177/1094428109333339     Document Type: Article

References (113)

1. Abelson, R.P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test. Psychological Science, 8, 12-15.
2. Aguinis, H. (2004). Regression analysis for categorical moderators. New York: Guilford.
3. Aguinis, H., Beaty, J.C., Boik, R.J., & Pierce, C.A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology, 90, 94-107.
4. Aguinis, H., & Harden, E.E. (2009). Sample size rules of thumb: Evaluating three common practices. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in the organizational and social sciences (pp. 269-288). New York: Routledge.
5. Aguinis, H., & Harden, E.E. (IN PRESS). Cautionary note on conveniently dismissing chi-square goodness-of-fit test results: Implications for strategic management research. In D. J. Ketchen & D. A. Bergh (Eds.), Research methodology in strategy and management (vol. 5). San Diego, CA: Elsevier.
6. Aguinis, H., Mazurkiewicz, M.D., & Heggestad, E.D. (2009). Using Web-based frame-of-reference training to decrease biases in personality-based job analysis: An experimental field study. Personnel Psychology, 62, 405-438.
7. Aguinis, H., Pierce, C.A., Bosco, F.A., & Muslin, I.S. (2009). First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods, 12, 69-112.
8. Aguinis, H., Pierce, C.A., & Culpepper, S.A. (IN PRESS). Scale coarseness as a methodological artifact: Correcting correlation coefficients attenuated from using coarse scales. Organizational Research Methods.
9. Aguinis, H., & Smith, M.A. (2007). Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. Personnel Psychology, 60, 165-199.
10. Aguinis, H., & Stone-Romero, E.F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82, 192-206.
11. Algina, J., & Keselman, H.J. (2003). Approximate confidence intervals for effect sizes. Educational and Psychological Measurement, 63, 537-553.
12. American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
13. Anderson-Levitt, K.M. (2006). Ethnography. In J. L. Green, G. Camilli & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 279-295). Mahwah, NJ: Lawrence Erlbaum.
14. Anderson, N. (2007). The practitioner-researcher divide revisited: Strategic-level bridges and the roles of IWO psychologists. Journal of Occupational and Organizational Psychology, 80, 175-183.
15. Armstrong, S.A., & Henson, R.K. (2004). Statistical and practical significance in the IJPT: A research review from 1993-2003. International Journal of Play Therapy, 13, 9-30.
16. Bacharach, S.B. (1989). Organizational theories: Some criteria for evaluation. Academy of Management Review, 14, 496-515.
17. Baldridge, D.C., Floyd, S.W., & Markóczy, L. (2004). Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strategic Management Journal, 25, 1063-1074.
18. Barnett, M.L. (2007). (Un)learning and (mis)education through the eyes of Bill Starbuck: An interview with Pandora's playmate. Academy of Management Learning and Education, 6, 114-127.
19. Bartunek, J.M. (2007). Academic-practitioner collaboration need not require joint or relevant research: Toward a relational scholarship of integration. Academy of Management Journal, 50, 1323-1333.
20. Bedeian, A.G., Sturman, M.C., & Streiner, D.L. (IN PRESS). Decimal dust, significant digits, and the search for stars. Organizational Research Methods.
21. Bedeian, A.G., Van Fleet, D.D., & Hyman, H.H., III. (2009). "Circle the wagons and defend the faith": Slicing and dicing the data. Organizational Research Methods, 12, 276-295.
22. Brews, P.J., & Tucci, C.L. (2004). Exploring the structural effects of internetworking. Strategic Management Journal, 25, 429-451.
23. Cascio, W.F., & Aguinis, H. (2001). The Federal Uniform Guidelines on Employee Selection Procedures (1978): An update on selected issues. Review of Public Personnel Administration, 21, 200-218.
24. Cascio, W.F., & Aguinis, H. (2008). Research in industrial and organizational psychology from 1963 to 2007: Changes, choices, and trends. Journal of Applied Psychology, 93, 1062-1081.
25. Cashen, L.H., & Geiger, S.W. (2004). Statistical power and the testing of null hypotheses: A review of contemporary management research and recommendations for future studies. Organizational Research Methods, 7, 151-167.
26. Chow, S.L. (1998). Précis of statistical significance: Rationale, validity, and utility. Behavioral and Brain Sciences, 21, 169-239.
27. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
28. Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). New York: Academic Press.
29. Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-253.
30. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
31. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
32. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
33. Cohen, J. (1994). The earth is round (p < 0.05). American Psychologist, 49, 997-1003.
34. Connelly, F.M., & Clandinin, D.J. (2006). Narrative inquiry. In J. L. Green, G. Camilli & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 477-488). Mahwah, NJ: Lawrence Erlbaum.
35. Cortina, J.M., & Folger, R.G. (1998). When is it acceptable to accept a null hypothesis: No way, Jose? Organizational Research Methods, 1, 334-350.
36. Cortina, J.M., & Landis, R.S. (2009). When small effect sizes tell a big story, and when large effect sizes don't. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Received doctrine, verity, and fable in the organizational and social sciences (pp. 287-308). New York: Routledge.
37. Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.
38. Cummings, T.G. (2007). 2006 Presidential address: Quest for an engaged academy. Academy of Management Review, 32, 355-360.
39. Duriau, V.J., Reger, R.K., & Pfarrer, M.D. (2007). A content analysis of the content analysis literature in organization studies: Research themes, data sources, and methodological refinements. Organizational Research Methods, 10, 5-34.
40. Easterby-Smith, M., Golden-Biddle, K., & Locke, K. (2008). Working with pluralism: Determining quality in qualitative research. Organizational Research Methods, 11, 419-429.
41. Fan, X. (2001). Statistical significance and effect size in education research: Two sides of a coin. Journal of Educational Research, 94, 275-282.
42. Fisher, R.A. (1925). Statistical methods for research workers. Edinburgh, Scotland: Oliver and Boyd.
43. Fisher, R.A. (1959). Statistical methods and scientific inference (2nd ed.). New York: Hafner.
44. Gelade, G.A. (2006). But what does it mean in practice? The Journal of Occupational and Organizational Psychology from a practitioner perspective. Journal of Occupational and Organizational Psychology, 79, 153-160.
45. Gephart, R.P. Jr. (2004). Qualitative research and the Academy of Management Journal. Academy of Management Journal, 47, 454-462.
46. Gigerenzer, G. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199-200.
47. Gilligan, C. (1982). In a different voice. Cambridge, MA: Harvard University Press.
48. Gómez-Bezares, F., & Gómez-Bezares, F.R. (2006). On the use of hypothesis testing and other problems in financial research. Journal of Investing, 15, 64-67.
49. Green, J. L., Camilli, G., & Elmore, P. B. (Eds.). (2006). Handbook of complementary methods in education research. Mahwah, NJ: Lawrence Erlbaum.
50. Griffin, C., & Bengry-Howell, A. (2008). Ethnography. In C. Willig & W. Sainton-Rogers (Eds.), The Sage handbook of qualitative research in psychology (pp. 15-31). Thousand Oaks, CA: Sage.
51. Hakel, M.D., Sorcher, M., Beer, M., & Moses, J.L. (1982). Making it happen-Designing research with implementation in mind. Beverly Hills, CA: Sage Publications.
52. Hambrick, D.C. (1994). 1993 Presidential Address: What if the Academy actually mattered? Academy of Management Review, 19, 11-16.
53. Hemphill, J.F. (2003). Interpreting the magnitude of correlation coefficients. American Psychologist, 58, 78-79.
54. Henkel, R.E. (1976). Tests of significance. Beverly Hills, CA: Sage Publications.
55. Hiles, D., & Čermák, I. (2008). Narrative psychology. In C. Willig & W. Sainton-Rogers (Eds.), The Sage handbook of qualitative research in psychology (pp. 147-164). Thousand Oaks, CA: Sage Publications.
56. Hillman, A. (2009). Editor's comments. Academy of Management Review, 34, 7-10.
57. Hubbard, R., & Ryan, P.A. (2000). The historical grown of statistical significance testing in psychology-and its future prospects. Educational and Psychological Measurement, 60, 661-681.
58. Huberty, C.J. (1993). Historical origins of statistical testing practices: The treatment of Fisher versus NeymanPearson views in textbooks. Journal of Experimental Education, 61, 317-333.
59. Huberty, C.J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227-240.
60. Hunter, J.E. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3-7.
61. Iacobucci, D. (2005). From the editor. Journal of Consumer Research, 32, 1-6.
62. Kim, H., Hoskisson, R.E., & Wan, W.P. (2004). Power dependence, diversification strategy, and performance in keiretsu member firms. Strategic Management Journal, 25, 613-636.
63. Kirk, R.E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
64. Kline, R. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
65. Koontz, H. (1961). The management theory jungle. Journal of the Academy of Management, 4, 174-188.
66. Krueger, L.E. (1998). The Ego has landed!: The 0.05 level of statistical significance is soft (Fisher) rather than hard (Neyman/Pearson). Behavioral and Brain Sciences, 21, 207-208.
67. Latham, G.P. (2007). A speculative perspective on the transfer of behavior science findings to the workplace: "The times they are a-changin. " Academy of Management Journal, 50, 1027-1032.
68. Little, J. (2001). Understanding statistical significance: A conceptual history. Journal of Technical Writing and Communication, 31, 363-372.
69. McCloskey, D.N. (1985). The loss function has been mislaid: The rhetoric of significance tests. American Economic Review, 75, 201-205.
70. Miller, D.J., Fern, M.J., & Cardinal, L.B. (2007). The use of knowledge for technological innovation within diversified firms. Academy of Management Journal, 50, 308-326.
71. Morgeson, F.P., & Campion, M.A. (2002). Minimizing tradeoffs when redesigning work: Evidence from a longitudinal quasi-experiment. Personnel Psychology, 55, 589-612.
72. Murphy, K.R., & Myors, B. (1998). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. Mahwah, NJ: Erlbaum.
73. Phillips, N., Sewell, G., & Jaynes, S. (2008). Applying critical discourse analysis in strategic management research. Organizational Research Methods, 11, 770-789.
74. Pierce, C.A., Block, R.A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values from multi-factor ANOVA designs. Educational and Psychological Measurement, 64, 916-924.
75. Podsakoff, P.M., & Dalton, D.R. (1987). Research methodology in organizational studies. Journal of Management, 13, 419-441.
76. Pratt, M.G. (2008). Fitting oval pegs into round holes: Tensions in evaluating and publishing qualitative research in top-tier North American journals. Organizational Research Methods, 11, 481-509.
77. Raver, J.L., & Gelfand, M.J. (2005). Beyond the individual victim: Linking sexual harassment, team processes, and team performance. Academy of Management Journal, 48, 387-400.
78. Rosnow, R.L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
79. Rynes, S.L. (2007). Let's create a tipping point: What academics and practitioners can do, alone and together. Academy of Management Journal, 50, 1046-1054.
80. Rynes, S.L., Colbert, A.E., & Brown, K.G. (2002). HR professionals' beliefs about effective human resource practices: Correspondence between research and practice. Human Resource Management, 41, 149-174.
81. Rynes, S.L., Giluk, T.L., & Brown, K.G. (2007). The very separate worlds of academic and practitioner periodicals in human resource management: Implications for evidence-based management. Academy of Management Journal, 50, 987-1008.
82. Sauley, K.S. & Bedeian, A.G. (1989). 05: A case of the tail wagging the distribution. Journal of Management, 15, 335-344.
83. Savall, H., Zardet, V., Bonnet, M., & Péron, M. (2008). The emergence of implicit criteria actually used by reviewers of qualitative research articles: Case of a European journal. Organizational Research Methods, 11, 510-540.
84. Sawyer, A.G., & Peter, J.P. (1983). The significance of statistical significance tests in marketing research. Journal of Marketing Research, 20, 122-133.
85. Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1, 115-129.
86. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.
87. Shapiro, D.L., Kirkman, B.L., & Courtney, H.G. (2007). Perceived causes and solutions of the translation problem in management research. Academy of Management Journal, 50, 249-266.
88. Shrout, P.E. (1997). Should significance tests be banned? Introduction to a special section exploring the pros and cons. Psychological Science, 8, 1-2.
89. Sink, C.A., & Stroh, H.R. (2006). Practical significance: The use of effect sizes in school counseling research. Professional School Counseling, 9, 401-411.
90. Smiddy, H.F. (1962). Presidential address: Theory and research for improvement of management practise. Academy of Management Proceedings (pp. 179-196). Briar Cliff Manor, NY: Academy of Management.
91. Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.
92. Special research forum: Knowledge transfer between academics and practitioners. (2001). Academy of Management Journal, 44, 340-440.
93. Steier, F. (Ed.). (1991). Research and reflexivity. London: Sage Publications.
94. Starbuck, W.H., Aguinis, H., Konrad, A.M., & Baruch, Y. (2008). Tradeoffs among editorial goals in complex publishing environments. In Y. Baruch, A. M. Konrad, H. Aguinis & W. H. Starbuck (Eds.), Opening the black box of editorship (pp. 250-270). New York: Palgrave-Macmillan.
95. Thompson, B. (1999a). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
96. Thompson, B. (1999b). Statistical significance tests, effect size reporting and the vain pursuit of pseudo-objectivity. Theory and Psychology, 9, 191-196.
97. Thompson, B. (2002). "Statistical," "practical," and "clinical:" How many kinds of significance do counselors need to consider? Journal of Counseling & Development, 80, 64-71.
98. Thompson, B. (2006). Role of effect sizes in contemporary research in counseling. Counseling and Values, 50, 176-186.
99. Trieschmann, J.S., Dennis, A.R., Northcraft, G.B., & Niemi, A.W. Jr. (2000). Serving multiple constituencies in business schools: M.B.A. program versus research performance. Academy of Management Journal, 43, 1130-1141.
100. Tryon, W.W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386.
101. Tukey, J.W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100-116.
102. Tushman, M., & O'Reilly, C. (2007). Research and Relevance: Implications of Pasteur's Quadrant for Doctoral Programs and Faculty Development. Academy of Management Journal, 50, 769-774.
103. Vacha-Haase, T., Nilsson, J.E., Reetz, D.R., Lance, T.S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
104. Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
105. Vaske, J.J., Gliner, J.A., & Morgan, G.A. (2002). Communicating judgements about practical significance: Effect size, confidence intervals and odds ratios. Human Dimensions of Wildlife, 7, 287-300.
106. Vermeulen, F. (2007). I shall not remain insignificant: Adding a second loop to matter more. Academy of Management Journal, 50, 754-761.
107. Welcome to the Academy of Management Journal: Mission statement. (2009). Academy of Management Journal. Retrieved January 26, 2009, from http://journals.aomonline.org/amj/.
108. Werner, S., & Brouthers, L.E. (2002). How international is management? Journal of International Business Studies, 33, 583-591.
109. Wilkinson, S., & Kitzinger, C. (2006). Surprise as an interactional achievement: Reaction tokens in conversation. Social Psychology Quarterly, 69, 150-182.
110. Wilkinson, S., & Kitzinger, C. (2008). Conversation analysis. In C. Willig & W. Sainton-Rogers (Eds.), The Sage handbook of qualitative research in psychology (pp. 54-71). Thousand Oaks, CA: Sage Publications.
111. Willig, C., & Stainton-Rogers, W. (2008). Introduction. In C. Willig & W. Sainton-Rogers (Eds.), The Sage handbook of qualitative research in psychology (pp. 1-12). Thousand Oaks, CA: Sage Publications.
112. Yates, F. (1951). The influence of "statistical methods for research workers" on the development of the science of statistics. Journal of the American Statistical Association, 46, 19-34.
113. Zhang, X., Bartol, K.M., Smith, K.G., Pfarrer, M.D., & Khanin, D.M. (2008). CEOs on the edge: Earnings manipulation and stock-based incentive misalignment. Academy of Management Journal, 51, 241-258.