What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research

Journal of International Business Studies

Volumn 48, Issue 5, 2017, Pages 535-551

  Meyer, Klaus E ^a   Van Witteloostuijn, Arjen ^b,c,d   Beugelsdijk, Sjoerd ^e  

^a CHINA EUROPE INTERNATIONAL BUSINESS SCHOOL   (China)

^b TILBURG UNIVERSITY   (Netherlands)

^c ANTWERP MANAGEMENT SCHOOL   (Belgium)

^d CARDIFF UNIVERSITY   (United Kingdom)

^e UNIVERSITY OF GRONINGEN   (Netherlands)

Author keywords

hypothesis testing; new guidelines; p value; research malpractices; statistical reporting

Indexed keywords

EID: 85019894740     PISSN: 00472506     EISSN: 14786990     Source Type: Journal    
DOI: 10.1057/s41267-017-0078-8     Document Type: Review

References (88)

1. Aguinis, H., Werner, S., Abbott, J. L., Angert, C., Park, J. H., & Kohlhausen, D. 2010. Customer-centric research: reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods, 13(3): 515-539.
2. Andersson, U., Cuervo-Cazurra, A., & Nielsen, B. B. 2014. Explaining interaction effects within and across levels of analysis. Journal of International Business Studies, 45(9): 1063-1071.
3. Angrist, J. D., & Krueger, A. 2001. Instrumental variables and the search for identification: form supply and demand to natural experiments. Journal of Economic Perspectives, 15(4): 69-85.
4. Angrist, J. D., & Pischke, J. S. 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24(2): 3-30.
5. Antonakis, J., Bendahan, S., Jacquart, P., & Lalive, R. 2010. On making causal claims: A review and recommendations. Leadership Quarterly, 21(6): 1086-1120.
6. Barley, S. R. 2016. 60th anniversary essay: ruminations on how we became a mystery house and how we might get out. Administrative Science Quarterly, 61(1): 1-8.
7. Bedeian, A. G., Taylor, S. G., & Miller, A. 2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9(4): 715-725.
8. Bettis, R. A. 2012. The search for asterisks: compromised statistical tests and flawed theory. Strategic Management Journal, 33(1): 108-113.
9. Bettis, R. A., Ethiraj, S., Gambardella, A., Helfat, C. E., & Mitchell, W. 2016a. Creating repeatable cumulative knowledge in strategic management. Strategic Management Journal, 37(2): 257-261.
10. Bettis, R. A., Helfat, C. E., & Shaver, M. J. 2016b. Special issue: replication in strategic management. Strategic Management Journal, 37(11): 2191-2388.
11. Beugelsdijk, S., de Groot, H. L. F., & van Schaik, A. B. T. M. 2004. Trust and economic growth: A robustness analysis. Oxford Economic Papers, 56(1): 118-134.
12. Beugelsdijk, S., Slangen, A., Onrust, M., van Hoorn, A., & Maseland, R. 2014. The impact of home-host cultural distance on foreign affiliate sales: The moderating role of cultural variation within host countries. Journal of Business Research, 67(8): 1638-1646.
13. Bhattacharjee Y. 2013. The mind of a con man. New York Times Magazine, April 26.
14. Bobko, P. 2001. Correlation and regression: Applications for industrial organizational psychology and management (2nd ed.). Thousand Oaks, CA: Sage.
15. Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. 2016. HARKing's threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69(3): 709-750.
16. Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. 2015. Correlational effect size benchmarks. Journal of Applied Psychology, 100(2): 431-449.
17. Brambor, T., Clark, W. R., & Golder, M. 2006. Understanding interaction models: improving empirical analyses. Political Analysis, 14(1): 63-82.
18. Branch, M. 2014. Malignant side-effects of null-hypothesis testing. Theory and Psychology, 24(2): 256-277.
19. Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. 2016. Star wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1): 1-32.
20. Buckley, P., Devinney, T., & Louviere, J. J. 2007. Do managers behave the way theory suggests? A choice-theoretic examination of foreign direct investment location decision-making. Journal of International Business Studies, 38(7): 1069-1094.
21. Cascio, W. F., & Zedeck, S. 1983. Open a new window in rational research planning: Adjust alpha to maximize statistical power. Personnel Psychology, 36(3): 517-526.
22. Choi, J., & Contractor, F. 2016. Choosing an appropriate alliance governance mode: The role of institutional, cultural and geographic distance in international research & development (R&D) collaborations. Journal of International Business Studies, 47(2): 210-232.
23. Cohen, J. 1969. Statistical power analysis for the behavioral sciences. New York: Academic Press.
24. Cortina, J. M., Köhler, T., & Nielsen, B. B. 2015. Restriction of variance interaction effects and their importance for international business. Journal of International Business Studies, 46(8): 879-885.
25. Crosswell, J. M., et al. 2009. Cumulative incidence of false positive results in repeated, multimodal cancer screening. Annals of Family Medicine, 7(3): 212-222.
26. Dalton, D. R., Aguinis, H., Dalton, C. A., Bosco, F. A., & Pierce, C. A. 2012. Revisiting the file drawer problem in meta-analysis: An empirical assessment of published and non-published correlation matrices. Personnel Psychology, 65(2): 221-249.
27. Dikova, D., Parker, S. C., & van Witteloostuijn, A. 2017. Capability, environment and internationalization fit, and financial and marketing performance of MNEs' foreign subsidiaries: An abductive contingency approach, Cross-Cultural and Strategic Management (forthcoming).
28. Doh, J. 2015. Why we need phenomenon-based research in international business. Journal of World Business, 50(4): 609-611.
29. Doucouliagos, C., & Stanley, T. D. 2013. Are all economic facts greatly exaggerated? Theory competition and selectivity. Journal of Economic Surveys, 27(2): 316-339.
30. Economist. 2014. When science gets it wrong: Let the light shine in. June 14. http://www.economist.com/news/scienceand-technology/21604089-two-big-recent-scientific-resultsare-looking-shakyand-it-open-peer-review (accessed March 23, 2017).
31. Ferguson, C. J., & Heene, M. 2012. A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspectives on Psychological Science, 7(6): 555-561.
32. Fisher, R. A. 1925. Statistical methods for research workers. Edinburgh: Oliver and Boyd.
33. Fisher, R., & Schwartz, S. 2011. Whence differences in value priorities? Individual, cultural, and artefactual sources. Journal of Cross-Cultural Psychology, 42(7): 1127-1144.
34. Fox, P. J., & Glas, C. A. W. 2002. Modeling measurement error in a structural multilevel model. In G. A. Marcoulides & I. Moustaki (Eds.), Latent variable and latent structure models. London: Lawrence Erlbaum Associates.
35. Gerber, A. S., Green, D. P., & Nickerson, D. 2001. Testing for publication bias in political science. Political Analysis, 9(4): 385-392.
36. Gigerenzer, G. 2004. Mindless statistics. Journal of Socio-Economics, 33(5): 587-606.
37. Goldfarb, B., & King, A. 2016. Scientific apophenia in strategic management research: Significance tests & mistaken inference. Strategic Management Journal, 37(1): 167-176.
38. Görg, H., & Strobl, E. 2001. Multinational companies and productivity spillovers: A meta-analysis with a test for publication bias. Economic Journal, 111: F723-F739.
39. Greene, W. 2010. Testing hypotheses about interaction terms in nonlinear models. Economics Letters, 107: 291-296.
40. Grieneisen, M. L., & Zhang, M. 2012. A comprehensive survey of retracted articles from the scholarly literature. PLoS ONE, 7(10): e44118. doi:10.1371/journal.pone.0044118.
41. Haans, R. F. P., Pieters, C., & He, Z. L. 2016. Thinking about U: Theorizing and testing U-and inverted U-shaped relationships in strategy research. Strategic Management Journal, 37(7): 1177-1196.
42. Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. 2015. The extent and consequences of p-hacking in science. PLoS Biology, 13(3): e1002106. doi:10.1371/journal. pbio.1002106.
43. Henrich, J., Heine, S. J., & Norenzayan, A. 2010a. The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3): 61-83.
44. Henrich, J., Heine, S. J., & Norenzayan, A. 2010b. Most people are not WEIRD. Nature, 466: 29.
45. Hoetker, G. 2007. The use of logit and probit models in strategic management research: Critical issues. Strategic Management Journal, 28(4): 331-343.
46. Hubbard, R., Vetter, D. E., & Little, E. L. 1998. Replication in strategic management: scientific testing for validity, generalizability, and usefulness. Strategic Management Journal, 19(3): 243-254.
47. Hunter, J. E., & Schmidt, F. L. 2015. Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
48. Husted, B. W., Montiel, I., & Christmann, P. 2016. Effects of local legitimacy on certification decision to global and national CSR standards by multinational subsidiaries and domestic firms. Journal of International Business Studies, 47(3): 382-397.
49. Ioannidis, J. P. A. 2005. Why most published research findings are false. PLoS Medicine, 2(8): e124.
50. Ioannidis, J. P. A. 2012. Why science is not necessarily selfcorrecting. Perspectives on Psychological Science, 7(6): 645-654.
51. John, L. K., Loewenstein, G., & Prelec, D. 2012. Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23(5): 524-532.
52. Kerr, N. L. 1998. HARKIng: Hypothesizing after results are known. Personality and Social Psychology Review, 2(3): 196-217.
53. Kingsley, A. F., Noordewier, T. G., & Vanden Bergh, R. G. 2017. Overstating and understating interaction results in international business research. Journal of World Business, 52(2): 286-295.
54. Kirk, R. E. 1996. Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5): 746-759.
55. Leamer, E. E. 1985. Sensitivity analyses would help. American Economic Review, 75(3): 308-313.
56. Lewin, A. Y., Chiu, C. Y., Fey, C. F., Levine, S. S., McDermott, G., Murmann, J. P., & Tsang, E. 2016. The critique of empirical social science: New policies at Management and Organization Review. Management and Organization Review, 12(4): 649-658.
57. Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. 2003. Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. British Medical Journal, 326(7400): 1167-1170.
58. Masicampo, E. J., & Lalande, D. R. 2012. A peculiar prevalence of p-values just below.05. Quarterly Journal of Experimental Psychology, 65(11): 2271-2279.
59. McCloskey, D. N. 1985. The loss function has been mislaid: the rhetoric of significance tests. American Economic Review, 75(2): 201-205.
60. McCloskey, D. N., & Ziliak, S. T. 1996. The standard error of regressions. Journal of Economic Literature, 34(March): 97-114.
61. Meyer, K. E. 2006. Asian management research needs more self-confidence. Asia Pacific Journal of Management, 23(2): 119-137.
62. Meyer, K. E. 2009. Motivating, testing, and publishing curvilinear effects in management research. Asia Pacific Journal of Management, 26(2): 187-193.
63. Misangyi, V. F., & Acharya, A. G. 2014. Substitutes or complements? A configurational examination of corporate governance mechanisms. Academy of Management Journal, 57(6): 1681-1705.
64. Mullane, K., & Williams, M. 2013. Bias in research: the rule rather than the exception? Elsevier Journal, http://editorsupdate. elsevier.com/issue-40-september-2013/bias-in-research-therule-rather-than-the-exception/(accessed March 23, 2017).
65. New York Times. 2011. Fraud case seen as a red flag for psychology research, November 2: http://www.nytimes.com/2011/11/03/health/research/noted-dutch-psychologist-stapelaccused-of-research-fraud.html?-r=1&ref=research (accessed January 15, 2017).
66. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science. doi:10.1126/science. aac4716.
67. Orlitzky, M. 2012. How can significance tests be deinstitutionalized? Organizational Research Methods, 15(2): 199-228.
68. Pashler, H., & Wagenmakers, E.-J. 2012. Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science, 7(6): 528-530.
69. Peterson, M., Arregle, J. L., & Martin, X. 2012. Multi-level models in international business research. Journal of International Business Studies, 43(5): 451-457.
70. Pfeffer, J. 2007. A modest proposal: How we might change the process and product of managerial research. Academy of Management Journal, 50(6): 1334-1345.
71. Popper, K. 1959. The logic of scientific discovery. London: Hutchinson.
72. Reeb, D., Sakakibara, M., & Mahmood, I. P. 2012. From the editors: Endogeneity in international business research. Journal of International Business Studies, 43(3): 211-218.
73. Rosenthal, R. 1979. The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86(3): 638-641.
74. Rosnow, R. L., & Rosenthal, R. 1984. Understanding behavioral science: Research methods for customers. New York:McGraw-Hill.
75. Rothstein, H. R., Sutton, A. J., & Borenstein, M. 2005. Publication bias in meta-analysis, prevention, assessment and adjustment. New York: Wiley.
76. Sala-i-Martin, X. 1997. I just ran two million regressions. American Economic Review, 87(2): 178-183.
77. Shadish, W. R., Cook, T. D., & Campbell, D. 2002. Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.
78. Simmons, J. P., Nelson, L. D., & Simonsohn, U. 2011. Falsepositive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11): 1359-1366.
79. Sterling, T. D. 1959. Publication decision and their possible effects on inferences drawn from tests of significance-vice versa. Journal of the American Statistical Association, 54(285): 30-34.
80. van Witteloostuijn, A. 2015. Toward experimental international business: Unraveling fundamental causal linkages. Cross-Cultural and Strategic Management, 22(4): 530-544.
81. van Witteloostuijn, A. 2016. What happened to Popperian falsification? Publishing neutral and negative findings. Cross Cultural & Strategic Management, 23(3): 481-508.
82. Wasserstein, R. L., & Lazar, N. A. 2016. The ASA's statement on p-values: Context, process, and purpose. American Statistician, 70(2): 129-133. http://amstat.tandfonline.com/doi/full/10. 1080/00031305.2016.1154108 (ASA = American Statistical Association).
83. Wiersema, M. F., & Bowen, H. P. 2009. The use of limited dependent variable techniques in strategy research: Issues and methods. Strategic Management Journal, 30(6): 679-692.
84. Williams, R. 2012. Using the margins command to estimate and interpret adjusted predictions and marginal effects. Stata Journal, 12(2): 308.
85. Wonnacott, T. H., & Wonnacott, R. J. 1990. Introductory statistics for business and economics. New York: Wiley Publishers.
86. Zedeck, S. 2003. Editorial. Journal of Applied Psychology, 88(1): 3-5.
87. Zellmer-Bruhn, M., Caligiuri, P., & Thomas, D. 2016. From the editors: Experimental designs in international business research. Journal of International Business Studies, 47(4): 399-407.
88. Zelner, B. 2009. Using simulation to interpret results from logit, probit, and other nonlinear models. Strategic Management Journal, 30(12): 1335-1348.