A call for openness in research reporting: How to turn covert practices into helpful tools

Academy of Management Learning and Education

Volumn 16, Issue 1, 2017, Pages 125-141

  Schwab, Andreas ^a   Starbuck, William H ^b,c  

^a IOWA STATE UNIVERSITY   (United States)

^b UNIVERSITY OF OREGON   (United States)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85018466939     PISSN: 1537260X     EISSN: None     Source Type: Journal    
DOI: 10.5465/amle.2016.0039     Document Type: Article

References (85)

1. Academy of Management. 2006. Code of Ethics. Retrieved from: Http://aom.org/uploadedFiles/About-AOM/Governance/AOM-Code-of-Ethics.pdf, Accessed February 15, 2017.
2. American Psychological Association. 2010. Publication manual of the American Psychological Association, (6th ed.). Washington, DC: American Psychological Association.
3. American Statistical Association Board of Directors. 2016. ASA statement on statistical significance and p-values. In Ronald L. Wasserstein, (Ed.), The American Statistician, 70(2): 131-133.
4. Anonymous. 2015. The case of the hypothesis that never was; Uncovering the deceptive use of post hoc hypotheses. Journal of Management Inquiry, 24: 214-216.
5. AOM Ethics Education Committee. 2011. Ethics video series: Reporting research. Retrieved from: Https://www.youtube.com/watch? v5fbFqLvuosbM&feature5related, Accessed January 30, 2016.
6. Armstrong, J. S., & Green, K. C. 2016. Guidance for science: Evidence and checklists. Retrieved from: Http://repository.upenn. edu/marketing-papers/181, Accessed February 16, 2017.
7. Aschwanden, C. 2015. Science isn't broken: It's just a hell of a lot harder than we give it credit for. FiveThirtyEight, Retrieved from: Http://fivethirtyeight.com/features/science-isnt-broken, Accessed August 21, 2015.
8. Ashby, W. R. 1956. The black box. An introduction to cybernetics: Chapter 6, 86-117. London: Chapman & Hall.
9. Bakker, M., &Wicherts, J. M. 2011. The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3): 666-678.
10. Banks, G. C., O'Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., & Adkins, C. L. 2016a. Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1): 5-20.
11. Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. 2016b. Editorial: Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31(3): 323-338.
12. Bedeian, A. G. 2003. The manuscript review process: The proper roles of authors, referees, and editors. Journal of Management Inquiry, 12(4): 331-338.
13. Bedeian, A. G., Sturman, M. C., & Streiner, D. L. 2009. Decimal dust, significant digits, and the search for stars. Organizational Research Methods, 12(4): 687-694.
14. Bedeian, A. G., Taylor, S. G., & Miller, A. N. 2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9(4): 715-725.
15. Bem, D. J. 2004. Writing the empirical journal article. In J. M. Darley, M. P. Zanna, & H. L. Roediger, III, (Eds.), The compleat academic: A practical guide for the beginning social scientist, (2nd ed.): 171-201. Washington, DC: American Psychological Association.
16. Bergh, D. D., Sharp, B., Aguinis, H., & Li, M. 2015. The looming replication crisis in strategic management. Presentation at the Annual International Conference of the Strategic Management Society, Denver, CO.
17. Bettis, R. A., Ethiraj, S., Gambardella, A., Helfat, C., & Mitchell, W. 2016. Creating repeatable cumulative knowledge in strategic management. Strategic Management Journal, 37(2): 257-261.
18. Bhattacharjee, Y. 2013. The mind of a con man. The New York Times (August 23). Retrieved from: Http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academicfraud. html?login5email&pagewanted5all&-r50, Accessed June 6, 2016.
19. Biemann, T. 2013. What if we were Texas sharpshooters? Predictor reporting bias in regression analysis. Organizational Research Methods, 16: 335-363.
20. Bobko, P., Roth, P. L., & Buster, M. A. 2007. The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta-analysis. Organizational Research Methods, 10: 689-709.
21. Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. 2016. HARKing's threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69: 709-750.
22. Bunge, M. 1963. A general black box theory. Philosophy of Science, 30(4): 346-358.
23. Byington, E. K., & Felps, W. 2017. Solutions to the credibility crisis in management science. Academy of Management Learning & Education, 16(1): 142-162.
24. Chang, A. C., & Li, P. 2015. Is economics research replicable? Sixty published papers from thirteen journals say 'usually not'. Finance and Economics Discussion Series 2015-083. Washington, DC: Board ofGovernors of the Federal Reserve System.
25. Cochran,W. G. 1977. Sampling techniques, (3rd ed.).NewYork: Wiley.
26. Cumming, G. 2011. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
27. Einhorn, H. J., & Hogarth, R. M. 1975. Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13: 171-192.
28. Fanelli, D. 2013. Redefine misconduct as distorted reporting. Nature, 494: 149.
29. Fiedler, K., & Schwarz, N. 2016. Questionable research practices revisited. Social Psychological & Personality Science, 7(1): 45-52.
30. Fidler, F. 2005. From statistical significance to effect estimation: Statistical reform in psychology, medicine and ecology. Doctoral dissertation, University of Melbourne. Downloadable at http://www.latrobe.edu.au/psy/staff/fidler.html
31. Folger, R., & Stein, C. M. 2016. Abduction 101: Reasoning processes to aid discovery. Human Resources Management Review. Retrieved from: Http://dx.doi.org/10.1016/j.hrmr.2016.08.007. Available online August 30, 2016.
32. Francis, G., Tanzman, J., & Matthews, W. J. 2014. Excess success for psychology articles in the journal Science. PLoS One, 9(12): E114255.
33. Gauch, H. G., Jr., 2006. Winning the accuracy game. American Scientist, 94: 135-143.
34. Goldfarb, B., & King, A. A. 2015. Scientific apophenia in strategic management research: Significance tests & mistaken inference. Strategic Management Journal, 37: 167-176.
35. Green, K. C., & Armstrong, J. S. 2015. Simple versus complex forecasting: The evidence. Journal of Business Research, 68: 1678-1685.
36. Greenwald, A. G. 1975. Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82: 1-20.
37. Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.) 1983. Understanding robust and exploratory data analysis. New York: Wiley.
38. Hodgkinson, G. P., & Starkey, K. 2012. Extending the foundations and reach of design science: Further reflections on the role of critical realism. British Journal of Management, 23(4): 605-610.
39. Honig, B., & Bedi, A. 2012. The fox in the hen house: A critical examination of plagiarism among members of the Academy of Management. Academy of Management Learning & Education, 11(1): 101-123.
40. Honig, B., Lampel, J., Siegel, D., & Drnevich, P. 2014. Ethics in the production and dissemination of management research: Institutional failure or individual fallibility? Journal of Management Studies, 51(1): 118-142.
41. Hubbard, R., & Lindsay, R. M. 2013. The significant difference paradigm promotes bad science. Journal of Business Research, 66: 1393-1397.
42. Hubbard, R. 2015. Corrupt research: The case for reconceptualizing empirical management and social science. Los Angeles: SAGE Publications.
43. Jensen, N. M. 2015. Data access and replication policies in management. Retrieved from: Http://www.natemjensen.com/2015/11/10/data-access-and-replication-policies-in-management/; Accessed June 6, 2016.
44. John, L. K., Loewenstein, G., & Prelec, D. 2012. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5): 524-532.
45. Journal of Business and Psychology. 2016. JBP Special initiative hybrid registered reports resource site. Retrieved from: Https://jbp.uncc.edu/, Accessed January 30, 2016.
46. Kagle, A. 2008. Driven to settle: Eliot Spitzer v. GlaxoSmithKline and undisclosed clinical trials data regarding Paxil. (August 18, 2008). Retrieved from:http://web.law.columbia.edu/sites/default/files/microsites/career-services/Driven%20to%20Settle. pdf
47. Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. 2012. Publication bias in the organizational sciences. Organizational Research Methods, 15(4): 624-662.
48. Kepes, S., Bennett, A. A., & McDaniel, M. A. 2014. Evidence-based management and the trustworthiness of our cumulative scientific knowledge: Implications for teaching, research, and practice. Academy of Management Learning & Education, 13(3): 446-466.
49. Kepes, S., & McDaniel, M. A. 2013. How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(3): 252-268.
50. Kerr, N. L. 1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3): 196-217.
51. Knaub, J. 2008. Finite population correction (FPC) factor. In P. Lavrakas (Ed.), Encyclopedia of survey research methods: 284-286.Thousand Oaks, CA: SAGE.
52. Landis, R. S., & Rogelberg, S.G. 2013.Our scholarly practices are derailing our progress: The importance of "nothing" in the organizational sciences. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(3): 299-302.
53. LeBel, E. P., Campbell, L., & Loving, T. J. In press. Benefits of open and high-powered research outweigh costs. Journal of Personality and Social Psychology. Retrieved from: Http://papers.ssrn.com/sol3/papers.cfm?abstract-id52616384, Accessed: February 3, 2017.
54. Locke, E. A. 2007. The case for inductive theory building. Journal of Management, 33: 867-890.
55. Lovell, M. C. 1983. Data mining. The Review of Economics and Statistics, 65: 1-12.
56. Mazzola, J. J., & Deuling, J. K. 2013. Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I-O journal articles. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6: 279-284.
57. McShane, B. B., & Gal, D. 2016. Blinding us to the obvious? The effect of statistical training on the evaluation of evidence. Management Science, 62(6): 1707-1718.
58. McWilliams, A., Siegel, D., & Teoh, S. H. 1999. Issues in the use of the event study methodology:Acritical analysis of corporate social responsibility studies. Organizational Research Methods, 2: 340-365.
59. Merton, R. K. 1973. The normative structure of science. In N. Storer (Ed.), The sociology of science: Theoretical and empirical investigations: 267-278. Chicago, IL: University of Chicago Press.
60. Miller, A. N., Taylor, S. G., & Bedeian, A. G. 2011. Publish or perish: Academic life as management faculty live it. Career Development International, 16(5): 422-445.
61. O'Boyle, E. H., Banks, G. C., & Gonzalez-Mul é, E. 2014. The Chrysalis Effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2): 376-399.
62. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science, 349(6251): 943, aaac4716.
63. Orlitzky, M. 2012. How can significance tests be deinstitutionalized? Organizational Research Methods, 15(2): 199-228.
64. Peach, J. T., & Webb, J. L. 1983. Randomly specified macroeconomic models: Some implications for model selection. Journal of Economic Issues, 17(3): 697-720.
65. Pfeffer, J. 1993. Barriers to the advance of organizational science: Paradigm development as a dependent variable. Academy of Management Review, 18(4): 599-620.
66. Rosenthal, R. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin, 86: 638-641.
67. Rousseeuw, P. J., &Leroy, A. M. 1987. Robust regression and outlier detection. New York: Wiley.
68. Schmidt, F. L., & Hunter, J. E. 2014. Methods of meta-analysis: Correcting error and bias in research findings. Newberry Park, CA: SAGE.
69. Schwab, A., Abrahamson, E., Fidler, F., & Starbuck, W. H. 2011. Researchers should make thoughtful assessments instead of null-hypothesis significance tests. Organization Science, 22(4): 1105-1120.
70. Schwab, A., & Starbuck, W. H. 2016. Collegial "nests" can foster critical thinking, innovative ideas, and scientific progress. Strategic Organization, 14(2): 167-177.
71. Selvin, H. C., & Stuart, A. 1966. Data-dredging procedures in survey analysis. The American Statistician, 20(3): 20-23.
72. Simonsohn, U. 2016. Evaluating replications: 40% full - 60% empty. Retrieved from: DataColada.org/47, Accessed March 03, 2016.
73. Sijtsma, K., Veldkamp, C. L., &Wicherts, J. M. 2015. Improving the conduct and reporting of statistical analysis in psychology. Psychometrika, 81(1): 33-38.
74. Silberzahn, R., & Uhlmann, E. L. 2015. Crowdsourced research: Many hands make tight work. Nature, 526(7572): 189-191, http://www.nature.com/news/crowdsourced-research-manyhands- make-tight-work-1.18508.
75. Siler, K., & Strang, D. 2016. Peer review and scholarly originality: Let 1,000 flowers bloom, but don't step on any. Science, Technology and Human Values, online before print, DOI 10.1177/0162243916656919.
76. Simmons, J. P., Nelson, L. D., & Simonsohn, U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11): 1359-1366.
77. Starbuck, W. H. 2016a. 60th anniversary essay: How journals could improve research practices in social science. Administrative Science Quarterly, DOI: 0001839216629644.
78. Starbuck, W. H. 2016b. Squeezing lemons to make fresh lemonade: How to extract useful value from peer reviews. In T. Clark, M. Wright, & D. Ketchen (Eds.), How to get published in the best management journals: 85-108 Cheltenham: Edward Elgar.
79. Starbuck, W. H., & Dutton, J. M. 1971. The history of simulation models. In J. M. Dutton & W. H. Starbuck (Eds.), Computer simulation of human behavior: 9-102. New York: Wiley.
80. Sterling, T.D. 1959. Publication decisions and their possible effects on inferences drawn fromtests of significance-or vice versa. Journal of the American Statistical Association, 54: 30-34.
81. Tukey, J. W. 1977. Exploratory data analysis. Reading, CA: Addison-Wesley.
82. Wasserstein, R. L., & Lazar, N. A. 2016. The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70(2): 129-131.
83. Webster, J., & Starbuck, W. H. 1988. Theory building in industrial and organizational psychology. International Review of Industrial and Organizational Psychology, 3(1): 93-138.
84. Wigboldus, D. H., & Dotsch, R. 2016. Encourage playing with data and discourage questionable reporting practices. Psychometrika, 81: 27-32.
85. Woo, S. E., O'Boyle, E., & Spector, P. E. 2016. Best practices in developing, conducting, and evaluating inductive research. Human Resources Management Review, Retrieved from: Http://dx.doi.org/10.1016/j.hrmr.2016.08.004, Available online August 22, 2016.