Reproducibility in psychological science: When do psychological phenomena exist?

Frontiers in Psychology

Volumn 8, Issue JUN, 2017, Pages

  Iso Ahola, Seppo E ^a  

^a UNIVERSITY OF MARYLAND   (United States)

Author keywords

Ego depletion; Meta analysis; Psychological phenomena; Replication; Reproducibility

Indexed keywords

EID: 85020463768     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2017.00879     Document Type: Article

References (96)

1. Ackerman, P. (2014). Nonsense, common sense, and science of expert performance: talent and individual differences. Intelligence 45, 6-17. doi: 10.1016/j.intell.2013.04.009
2. Anderson, C., Bahník,Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., et al. (2016). Response to comment on "Estimating the reproducibility of psychological science". Science 351:1037.
3. Bandura, A. (1973). Aggression: A Social Learning Analysis . Englewood Cliffs, NJ: Prentice-Hall.
4. Bargh, J. (2014a). Our unconscious mind. Sci. Am. 310, 30-37. doi: 10.1038/scientificamerican0114-30
5. Bargh, J. (2014b). "The historical origins of priming as the preparation of behavioral responses: unconscious carryover and contextual influences of real-life importance," in Understanding Priming Effects in Social Psychology , ed. D. Molden (New York, NY: The Guildford Press), 218-233.
6. Bargh, J., Chen, M., and Burrows, L. (1996). Automaticity of social behavior: direct effects of trait construct and stereotype activation on action. J. Pers. Soc. Psychol. 71, 230-244. doi: 10.1037/0022-3514.71.2.230
7. Baron, R., and Kenny, D. (1986). The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J. Pers. Soc. Psychol. 51, 1173-1182. doi: 10.1037/0022-3514.51.6.1173
8. Bar-Yam, Y. (2016). The limits of phenomenology: from behaviorism to drug testing and engineering design. Complexity 21, 181-189. doi: 10.1002/cplx.21730
9. Baumeister, R., Bratslavsky, E., Muraven, M., and Tice, D. (1998). Ego depletion: Is the active self a limited resource? J. Pers. Soc. Psychol. 74, 1252-1265. doi: 10.1037/0022-3514.74.5.1252
10. Baumeister, R., and Heatherton, T. (1996). Self-regulation failure: an overview. Psychol. Inq. 7, 1-15. doi: 10.1207/s15327965pli0701_1
11. Baumeister, R., Masicampo, E., and Vohs, K. (2011). Do conscious thoughts cause behavior? Annu. Rev. Psychol. 62, 331-361. doi: 10.1146/annurev.psych.093008.131126
12. Baumeister, R., and Showers, C. (1986). A review of paradoxical performance effects: choking under pressure in sports and mental tests. Eur. J. Soc. Psychol. 16, 361-383. doi: 10.1002/ejsp.2420160405
13. Baumeister, R., Vohs, K., and Funder, D. (2007). Psychology as the science of self-reports and finger movements: Whatever happened to actual behavior? Perspect. Psychol. Sci. 2, 396-403. doi: 10.1111/j.1745-6916.2007.00051.x
14. Begley, S. (1997). The science wars. Newsweek 129, 54-57.
15. Bradley, W., and Mannell, R. (1984). Sensitivity of intrinsic motivation to reward procedure instructions. Pers. Soc. Psychol. Bull. 10, 426-431. doi: 10.1177/0146167284103011
16. Braver, S., Thoemmes, F. J., and Rosenthal, R. (2014). Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9, 333-342. doi: 10.1177/1745691614529796
17. Brennan, A. (2012). "Necessary and sufficient conditions," in The Stanford Encyclopedia of Philosophy , ed. E. Zalta (Stanford, CA: Metaphysics Laboratory).
18. Carter, E., Kofler, L., Forster, D., and McCullough, M. (2015). A series of meta-analytic tests of the depletion effect: self-control does not seem to rely on a limited resource. J. Exp. Psychol. Gen. 144, 796-815. doi: 10.1037/xge0000083
19. Csikszentmihalyi, M. (1982). Toward a psychology of optimal experience. Rev. Pers. Soc. Psychol. 3, 13-36.
20. Cumming, G. (2014). The new statistics: why and how. Psychol. Sci. 25, 7-29. doi: 10.1177/0956797613504966
21. Cumming, J. (2008). Replication and p intervals. Perspect. Psychol. Sci. 3, 286-300. doi: 10.1111/j.1745-6924.2008.00079.x
22. Cunningham, M., and Baumeister, R. (2016). How to make nothing out of something: analyses of the impact of study sampling and statistical interpretation in misleading meta-analytic conclusions. Front. Psychol. 7:1639. doi: 10.3389/fpsyg.2016.01639
23. Custers, R., and Aarts, H. (2010). The unconscious will: how the pursuit of goals operates outside of conscious awareness. Science 329, 47-50. doi: 10.1126/science.1188595
24. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspect. Psychol. Sci. 6, 274-290. doi: 10.1177/1745691611406920
25. Dotson, C. (1973). "Analysis of change," in Exercise and Sport Sciences Reviews , Vol. 1, ed. J. Wilmore (New York, NY: Academic Press), 393-419.
26. Dotson, C. (1980). Logic of questionable density. Res. Q. Exerc. Sport 51, 23-36. doi: 10.1080/02701367.1980.10609273
27. Doyen, S., Klein, O., Pichon, C.-L., and Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS ONE 7:e29081. doi: 10.1371/journal.pone.0029081
28. Earp, B., and Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6:621. doi: 10.3389/fpsyg.2015.00621
29. Ericsson, K., and Ward, P. (2007). Capturing the naturally occurring superior performance of experts in the laboratory: toward a science of expert and exceptional performance. Curr. Dir. Psychol. Sci. 16, 346-350. doi: 10.1111/j.1467-8721.2007.00533.x
30. Festinger, L. (1957). A Theory of Cognitive Dissonance. Stanford, CA: Stanford University Press.
31. Fisher, R. (1925). Statistical Methods for Research Workers. Edinburgh: Oliver & Boyd.
32. Francis, G. (2012). The psychology of replication and replication in psychology. Perspect. Psychol. Sci. 7, 585-594. doi: 10.1177/1745691612459520
33. Gelman, A., and Shalizi, C. (2013). Philosophy and the practice of Bayesian statistics. Br. J. Math. Stat. Psychol. 66, 8-38. doi: 10.1111/j.2044-8317.2011.02037.x
34. Gergen, K. (1973). Social psychology as history. J. Pers. Soc. Psychol. 26, 309-320. doi: 10.1037/h0034436
35. Gilbert, D., King, G., Pettigrew, S., and Wilson, T. (2016). Comment on "Estimating the reproducibility of psychological science". Science 351:1037. doi: 10.1126/science.aad7243
36. Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3-8. doi: 10.3102/0013189X005010003
37. Glass, G. (1977). Integrating findings: the meta-analysis of research. Rev. Res. Educ. 5, 351-379.
38. Goodman, S., Fanelli, D., and Ioannidis, J. (2016). What does reproducibility mean? Sci. Transl. Med. 8:341s12.
39. Hagger, M., Chatzisarantis, N., and Zwienenberg, M. (2016). A multilab preregistered replication of the ego depletion effect. Perspect. Psychol. Sci. 11, 546-573. doi: 10.1177/1745691616652873
40. Hagger, M., Wood, C., Stiff, C., and Chatzisarantis, N. (2010). Ego depletion and the strength model of self-control: a meta-analysis. Psychol. Bull. 136, 495-525. doi: 10.1037/a0019486
41. Haig, B. D. (2017). Tests of statistical significance made sound. Educ. Psychol. Meas. 77, 489-506. doi: 10.1177/0013164416667981
42. Hedges, L. (1981). Distribution theory for Glass's estimator of effect size and related estimators. J. Educ. Stat. 6, 107-128. doi: 10.2307/1164588
43. Hofmann, W., Vohs, K., and Baumeister, R. (2012). What people desire, feel conflicted about, and try to resist in everyday life. Psychol. Sci. 23, 582-588. doi: 10.1177/0956797612437426
44. Hubbard, T. (2015). The varieties of momentum-like experience. Psychol. Bull. 141, 1081-1119. doi: 10.1037/bul0000016
45. Hubbard, T. (2017a). Momentum in music: musical succession as physical motion. Psychomusicol. Music Mind Brain 27, 14-30. doi: 10.1037/pmu0000171
46. Hubbard, T. (2017b). Toward a general theory of momentum-like effects. Brain Process. doi: 10.1016/j.beproc.2017.02.019 [Epub ahead of print].
47. Hudson, N., Lucas, R., and Donnellan, M. (2017). Day-to-day affect is surprisingly stable: a 2- year longitudinal study of well-being. Soc. Psychol. Pers. Sci. 8, 45-54. doi: 10.1177/1948550616662129
48. Hunter, J., and Schmidt, F. (2004). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 2nd Edn. Thousand Oaks, CA: Sage. doi: 10.4135/9781412985031
49. Inzlicht, M., Gervais, W., and Berkman, E. (2016). Bias-Correction Techniques Alone Cannot Determine Whether Ego Depletion is Different from Zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015. Available at: http://dx.doi.org/10.2139/ssrn.2659409doi: 10.2139/ssrn.2659409
50. Inzlicht, M., and Schmeichel, B. (2012). What is ego depletion? Toward a mechanistic revision of the resource model of self-control. Perspect. Psychol. Sci. 7, 450-463. doi: 10.1177/1745691612454134
51. Iso-Ahola, S. (2013). Exercise: why it is a challenge for both the nonconscious and conscious mind. Rev. Gen. Psychol. 17, 93-110. doi: 10.1037/a0030657
52. Iso-Ahola, S. (2015). Conscious versus nonconscious mind and leisure. Leisure Sci. 37, 289-310. doi: 10.1080/01490400.2015.1005560
53. Iso-Ahola, S., and Dotson, C. (2014). Psychological momentum: why success breeds success. Rev. Gen. Psychol. 18, 19-33. doi: 10.1037/a0036406
54. Iso-Ahola, S., and Dotson, C. (2015). Psychological momentum-Not a statistical but psychological phenomenon. Rev. Gen. Psychol. 19, 112-116. doi: 10.1037/gpr0000026
55. Iso-Ahola, S., and Dotson, C. (2016). Psychological momentum- a key to continued success. Front. Psychol. 7:1328. doi: 10.3389/fpsyg.2016.01328
56. Iso-Ahola, S., and Dotson, C. (2017). Momentum and elite performance. J. Nature Sci. 3:e325.
57. Iso-Ahola, S., Dotson, C., Jagodinsky, A., Clark, L., Smallwood, L., Wilburn, C., et al. (2016). Improving performance by anchoring movement and "nerves". Hum. Mov. Sci. 49, 239-247. doi: 10.1016/j.humov.2016.07.008
58. Jones, L., and Stuth, G. (1997). The uses of mental imagery in athletics: an overview. Appl. Prev. Psychol. 6, 101-115. doi: 10.1016/S0962-1849(05)80016-2
59. Kahneman, D. (2011). Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux.
60. Kruschke, J. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspect. Psychol. Sci. 6, 299-312. doi: 10.1177/1745691611406925
61. Kuhn, T. (1962). The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.
62. Lakatos, I. (1971). "History of science and its rational reconstructions," in PSA 1970 Boston Studies in The Philosophy of Science , Vol. 8, eds R. Buck and R. Cohen (Dordrecht: Reidel), 91-135.
63. Lakens, D., and Evers, E. (2014). Sailing from the seas of chaos into the corridor of stability: practical recommendations to increase the informational value of studies. Pers. Psychol. Sci. 9, 278-292. doi: 10.1177/1745691614528520
64. LeBel, E., Berger, D., and Campbell, L. (2017). Falsifiability is Not Optional. Available at: osf.io/preprints/psyarxiv/dv94b
65. LeBel, E., and Peters, K. (2011). Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice. Rev. Gen. Psychol. 15, 371-379. doi: 10.1037/a0025172
66. Lee, N., Chatzisarantis, N., and Hagger, M. (2016). Adequacy of the sequential-task paradigm in evoking ego-depletion and how to improve detection of ego-depleting phenomena. Front. Psychol. 7:136. doi: 10.3389/fpsyg.2016.00136
67. Lewis, B., and Linder, D. (1997). Thinking about choking? Attentional processes and paradoxical performance. Pers. Soc. Psychol. Bull. 23, 937-944. doi: 10.1177/0146167297239003
68. Lykken, D. (1968). Statistical significance in psychological research. Psychol. Bull. 70, 151-159. doi: 10.1037/h0026141
69. Macnamara, B., Hambrick, D., and Oswald, F. (2014). Deliberate practice and performance in music, games, sports, education, and professions: a meta-analysis. Psychol. Sci. 25, 1608-1618. doi: 10.1177/0956797614535810
70. Malle, B. (2006). The actor-observer asymmetry in attribution: a (surprising) meta-analysis. Psychol. Bull. 132, 895-919. doi: 10.1037/0033-2909.132.6.895
71. Mani, A., Mullainathan, S., Shafer, E., and Zhao, J. (2013). Poverty impedes cognitive function. Science 341, 976-980. doi: 10.1126/science.1238041
72. Markman, K., and Guenther, C. (2007). Psychological momentum: intuitive physics and naïve beliefs. Pers. Soc. Psychol. Bull. 33, 800-812. doi: 10.1177/0146167207301026
73. Marsh, H., and Perry, C. (2005). Self-concept contributes to winning gold medals: causal ordering of self-concept and elite swimming performance. J. Sport Exerc. Psychol. 27, 71-91. doi: 10.1123/jsep.27.1.71
74. McFall, R. (1996). Making psychology incorruptible. Appl. Prev. Psychol. 5, 9-15. doi: 10.1016/S0962-1849(96)80021-7
75. Meehl, P. (1990). Appraising and amending theories: the strategy of Lakatosian defense and two principles that warrant it. Psychol. Inq. 1, 108-141. doi: 10.1207/s15327965pli0102_1
76. Meinz, E., and Hambrick, D. (2010). Deliberate practice is necessary but not sufficient to explain individual differences in piano sight-reading skill: the role of working memory capacity. Psychol. Sci. 21, 914-919. doi: 10.1177/0956797610373933
77. Molden, D. (ed.) (2014). Understanding Priming Effects in Social Psychology. New York, NY: The Guildford Press.
78. Open Science Collaboration (2015). Psychology. Estimating the reproducibility of psychological science. Science 349:aac4716. doi: 10.1126/science.aac4716
79. Oskarsson, A., Van Boven, L., McClelland, G., and Hastie, R. (2009). What's next? Judging sequences of binary events. Psychol. Bull. 135, 262-285. doi: 10.1037/a0014821
80. Platt, J. (1964). Strong inference. Science 146, 347-353. doi: 10.1126/science.146.3642.347
81. Popper, K. (1959). The Logic of Scientific Discovery. London: Hutchison.
82. Rohleder, N., Beulen, S., Chen, E., Wolf, J., and Kirschbaum, C. (2007). Stress on the dance floor: the cortisol stress response to social-evaluative threat in competitive ballroom dancers. Pers. Soc. Psychol. Bull. 33, 69-84. doi: 10.1177/0146167206293986
83. Rosenthal, R. (1984). Meta-Analytic Procedures for Social Research. Beverly Hills, CA: Sage.
84. Scheibehenne, B., Jamil, T., and Wagenmakers, E.-J. (2016). Bayesian evidence synthesis can reconcile seemingly inconsistent results: the case of hotel towel reuse. Psychol. Sci. 27, 1043-1046. doi: 10.1177/0956797616644081
85. Schmidt, F. (2010). Detecting and correcting the lies that data tell. Perspect. Psychol. Sci. 5, 233-242. doi: 10.1177/1745691610369339
86. Shadish, W., Cook, T., and Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York, NY: Houghton Miffin.
87. Simmons, J., Nelson, L., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632
88. Simons, D. (2014). The value of direct replication. Perspect. Psychol. Sci. 9, 76-80. doi: 10.1177/1745691613514755
89. Smedslund, J. (2016). Why psychology cannot be an empirical science. Integr. Psychol. Behav. Sci. 50, 185-195. doi: 10.1007/s12124-015-9339-x
90. Stanley, D., and Spence, J. (2014). Expectations of replications: Are yours realistic? Perspect. Psychol. Sci. 9, 305-318. doi: 10.1177/1745691614528518
91. Trafimow, D. (2003). Hypothesis testing and theory evaluation at the boundaries. Psychol. Rev. 110, 526-535. doi: 10.1037/0033-295X.110.3.526
92. Triplett, N. (1898). The dynamogenic factors in pacemaking and competition. Am. J. Psychol. 9, 507-533. doi: 10.2307/1412188
93. Vohs, K., Baumeister, R., and Ciarrocco, N. (2005). Self-regulation and self-presentation: regulatory resource depletion impairs impression management and effortful self-presentation depletes regulatory resources. J. Pers. Soc. Psychol. 88, 632-657. doi: 10.1037/0022-3514.88.4.632
94. Wegner, D. (1995). Ironic processes of mental control. Psychol. Rev. 101, 34-52. doi: 10.1037/0033-295X.101.1.34
95. Wetzels, R., Matzke, D., Lee, M., Rouder, J., Iverson, G., and Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: an empirical comparison using 855 t tests. Perspect. Psychol. Sci. 6, 291-298. doi: 10.1177/1745691611406923
96. Zajonc, R. (1965). Social facilitation. Science 149, 269-274. doi: 10.1126/science.149.3681.269