When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress

Review of General Psychology

Volumn 21, Issue 4, 2017, Pages 308-320

  Rubin, Mark ^a  

^a UNIVERSITY OF NEWCASTLE   (Australia)

Author keywords

Accommodation; Falsification; HARKing; Prediction; Replication crisis

Indexed keywords

EID: 85032296427     PISSN: 10892680     EISSN: 19391552     Source Type: Journal    
DOI: 10.1037/gpr0000128     Document Type: Article

References (68)

1. Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., & Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PLoS ONE, 12, e0172792. http://dx.doi.org/10.1371/journal.pone .0172792
2. Aguinis, H., Cascio, W. F., & Ramani, R. S. (2017). Science's reproducibility and replicability crisis: International business is not immune. Journal of International Business Studies, 48, 653-663. http://dx.doi .org/10.1057/s41267-017-0081-0
3. Baker, M. (2016, May 25). Is there a reproducibility crisis? Nature, 533, 452-454. http://dx.doi.org/10.1038/533452a
4. Banks, G. C., O'Boyle, E. H., Jr., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E.,. Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5-20. http://dx.doi.org/10 .1177/0149206315619011
5. Bedeian, A. G., Taylor, S. G., & Miller, A. N. (2010). Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education, 9, 715-725. http://dx .doi.org/10.5465/AMLE.2010.56659889
6. Bem, D. J. (1987). Writing the empirical journal. In M. P. Zanna & J. M. Darley (Eds.), The compleat academic: A practical guide for the beginning social scientist (pp. 171-201). Mahwah, NJ: Erlbaum.
7. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. http://dx.doi.org/10.1037/ a0021524
8. Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing's Threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709-750. http://dx.doi.org/10.1111/peps.12111
9. Butler, N., Delaney, H., & Spoelstra, S. (2017). The gray zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16, 94-109. http://dx.doi.org/10.5465/amle.2015.0201
10. Collins, R. (1994). Against the epistemic value of prediction over accommodation. Noûs, 28, 210-224. http://dx.doi.org/10.2307/2216049
11. de Groot, A. D. (2014). The meaning of "significance" for different types of research (E. J. Wagenmakers, D. Borsboom, J. Verhagen, R. Kievit, M. Bakker, A. Cramer,. H. L. J. van der Maas). Acta Psychologica, 148, 188-194. http://dx.doi.org/10.1016/j.actpsy.2014.02.001
12. Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78-89. http://dx.doi.org/10.1016/j.jmp .2015.10.003
13. Fanelli, D. (2010). Do pressures to publish increase scientists' bias? An empirical support from U.S. states' data. PLoS ONE, 5, e10271. http:// dx.doi.org/10.1371/journal.pone.0010271
14. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. Perspectives on Psychological Science, 7, 555-561. http://dx.doi.org/10 .1177/1745691612459059
15. Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7, 45-52. http://dx .doi.org/10.1177/1948550615612150
16. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606. http://dx.doi.org/10.1016/j.socec.2004.09.033
17. Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571. http://dx.doi.org/10 .1177/1745691612457576
18. Glaser, J., & Salovey, P. (1998). Affect in electoral politics. Personality and Social Psychology Review, 2, 156-172. http://dx.doi.org/10.1207/ s15327957pspr0203_1
19. Harker, D. (2006). Accommodation and prediction: The case of the persistent head. The British Journal for the Philosophy of Science, 57, 309-321. http://dx.doi.org/10.1093/bjps/axl004
20. Helgeson, V. S., & Fritz, H. L. (1998). A theory of unmitigated communion. Personality and Social Psychology Review, 2, 173-183. http://dx .doi.org/10.1207/s15327957pspr0203_2
21. Hitchcock, C., & Sober, E. (2004). Prediction versus accommodation and the risk of overfitting. The British Journal for the Philosophy of Science, 55, 1-34. http://dx.doi.org/10.1093/bjps/55.1.1
22. Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 5-18. http://dx.doi.org/10.1177/0149206316679487
23. Howson, C. (1988). Accommodation, prediction and Bayesian confirmation theory. Proceedings of the Biennial Meeting of the Philosophy of Science Association (Vol. 2, pp. 381-392). Philosophy of Science Association.
24. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/09567 97611430953
25. Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252-268. http://dx.doi.org/10.1111/iops.12045
26. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi .org/10.1207/s15327957pspr0203_4
27. Kerr, N. L. (2011). HARK! A herald sings. but who's listening? In R. M. Arkin (Ed.), Most underappreciated: 50 prominent social psychologists describe their most unloved work (pp. 126-131). New York, NY: Oxford University Press. http://dx.doi.org/10.1093/acprof:osobl/ 9780199778188.003.0024
28. Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In I. Lakatos & A. Musgrave (Eds.), Criticism and the growth of knowledge (pp. 96-191). London, England: Cambridge University Press.
29. Lange, M. (2001). The apparent superiority of prediction to accommodation as a side effect: A reply to Maher. British Journal for the Philosophy of Science, 52, 575-588.
30. Leung, K. (2011). Presenting post hoc hypotheses as a priori: Ethical and theoretical issues. Management and Organization Review, 7, 471-479. http://dx.doi.org/10.1017/CBO9781139171434.009
31. Lindsay, D. S., Simons, D. J., & Lilienfeld, S. O. (2016, December). Research Preregistration 101. APS Observer. Retrieved from https:// www.psychologicalscience.org/observer/research-preregistration-101
32. Matsunaga, M. (2007). Familywise error in multiple comparisons: Disentangling a knot through a critique of O'Keefe's arguments against alpha adjustment. Communication Methods and Measures, 1, 243-265. http:// dx.doi.org/10.1080/19312450701641409
33. Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? American Psychologist, 70, 487-498. http://dx.doi.org/10 .1037/a0039400
34. Mayo, D. G. (1991). Novel evidence and severe tests. Philosophy of Science, 58, 523-552. http://dx.doi.org/10.1086/289639
35. Mayo, D. G. (1996). Error and the growth of experimental knowledge. Chicago: University of Chicago Press. http://dx.doi.org/10.7208/chicago/ 9780226511993.001.0001
36. Mayo, D. G. (2008). How to discount double-counting when it counts: Some clarifications. The British Journal for the Philosophy of Science, 59, 857-879.
37. Mayo, D. G. (2010). An ad hoc save of a theory of adhocness? In D. G. Mayo & A. Spanos (Eds.), Error and inference: Recent exchanges on experimental reasoning, reliability, and the objectivity and rationality of science (pp. 155-169). New York, NY: Cambridge University Press.
38. Mayo, D. (2014). Some surprising facts about (the problem of) surprising facts (from the Dusseldorf Conference, February 2011). Studies in History and Philosophy of Science, 45, 79-86. http://dx.doi.org/10.1016/j.shpsa.2013 .10.005
39. Mayo, D. G., & Spanos, A. (2006). Severe testing as a basic concept in a Neyman-Pearson philosophy of induction. The British Journal for the Philosophy of Science, 57, 323-357. http://dx.doi.org/10.1093/bjps/axl003
40. Mazzola, J. J., & Deuling, J. K. (2013). Forgetting what we learned as graduate students: HARKing and selective outcome reporting in I-O journal articles. Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 279-284. http://dx.doi.org/10.1111/iops .12049
41. Motyl, M., Demos, A. P., Carsel, T. S., Hanson, B. E., Melton, Z. J., Mueller, A. B.,. Skitka, L. J. (2017). The state of social and personality science: Rotten to the core, not so bad, getting better, or getting worse? Journal of Personality and Social Psychology, 113, 34-58. http://dx.doi.org/10.1037/pspa0000084
42. Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P.,. Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. http://dx.doi .org/10.1038/s41562-016-0021
43. Murayama, K., Pekrun, R., & Fiedler, K. (2014). Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review, 18, 107-118. http://dx.doi.org/10.1177/108886 8313496330
44. Musgrave, A. (1974). Logical versus historical theories of confirmation. The British Journal for the Philosophy of Science, 25, 1-23. http://dx .doi.org/10.1093/bjps/25.1.1
45. Nosek, B. A., Ebersole, C. R., DeHaven, A., & Mellor, D. (2017). The preregistration revolution. Retrieved from osf.io/2dxu5
46. O'Boyle, E. H., Jr., Banks, G. C., & Gonzalez-Mulé, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 367-399.
47. Open Science Collaboration. (2015, August 28). Estimating the reproducibility of psychological science. Science, 349, aac4716. http://dx.doi.org/ 10.1126/science.aac4716
48. Parker, C. G. (2015). Reframing the reproducibility crisis: Using an error-statistical account to inform the interpretation of replication results in psychological research. Doctoral dissertation, Virginia Polytechnic Institute and State University. Retrieved from https://vtechworks.lib .vt.edu/handle/10919/52963
49. Popper, K. R. (1979). Objective knowledge: An evolutionary approach. Oxford: New York, NY: Oxford University Press.
50. Richards, T. (2016). HARKing Back: Lessons in investing from science. Retrieved from https://seekingalpha.com/article/3895286-harking-backlessons- investing-science
51. Rubin, M. (2017). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21, 269-275. http://dx.doi.org/10.1037/gpr0000123
52. Schlesinger, G. N. (1987). Accommodation and prediction. Australasian Journal of Philosophy, 65, 33- 42. http://dx.doi.org/10.1080/00048 408712342751
53. Schwab, A., & Starbuck, W. H. (2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning & Education, 16, 125-141. http://dx.doi.org/10 .5465/amle.2016.0039
54. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
55. Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017, April 17). Constraints on generality (COG): A proposed addition to all empirical papers. Retrieved from http://osf.io/preprints/psyarxiv/w9e3r
56. Stroebe, W. (2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134-144. http:// dx.doi.org/10.1016/j.jesp.2015.09.017 S?
57. Swiatkowski, W., & Dompnier, B. (2017). Replicability crisis in social psychology: Looking at the past to find new pathways for the future. Revue Internationale de Psychologie Sociale, 30, 111-124. http://dx.doi .org/10.5334/irsp.66
58. Szucs, D. (2016). A tutorial on hunting statistical significance by chasing N. Frontiers in Psychology, 7, 1444. http://dx.doi.org/10.3389/fpsyg.2016 .01444
59. Unkelbach, C. (2016). Increasing replicability. Social Psychology, 47, 1-3. http://dx.doi.org/10.1027/1864-9335/a000270
60. van 't Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology: A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2-12. http://dx.doi.org/10.1016/j.jesp .2016.03.004
61. Vazire, S. (2014). Life after Bem [Blog post]. Retrieved from http:// sometimesimwrong.typepad.com/wrong/2014/03/life-after-bem.html
62. Wagenmakers, E. J., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. (2015). A skeptical eye on psi. In E. C. May & S. B. Waraha (Eds.), Extrasensory perception: Support, skepticism, and science: (pp. 153-176). Santa Barbara, CA: Praeger.
63. Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H. L. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432. http://dx.doi.org/10.1037/a0022790
64. Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632-638. http://dx.doi.org/10 .1177/1745691612463078
65. Worrall, J. (1985). Scientific discovery and theory-confirmation. In J. C. Pitt (Ed.), Change and progress in modern science: Papers related to and arising from the Fourth International Conference on History and Philosophy of Science (pp. 301-331). Dordrecht, the Netherlands: Reidel. http://dx.doi.org/10.1007/978-94-009-6525-6_11
66. Worrall, J. (2010). Error, tests, and theory confirmation. In D. G. Mayo & A. Spanos (Eds.), Error and inference: Recent exchanges on experimental reasoning, reliability, and the objectivity and rationality of science (pp. 125-154). New York, NY: Cambridge University Press.
67. Worrall, J. (2014). Prediction and accommodation revisited. Studies in History and Philosophy of Science, 45, 54-61. http://dx.doi.org/10 .1016/j.shpsa.2013.10.001
68. Zahar, E. (1973). Why did Einstein's programme supersede Lorentz's? (I). The British Journal for the Philosophy of Science, 24, 95-123. http:// dx.doi.org/10.1093/bjps/24.2.95