Replicability crisis in social psychology: Looking at the past to find new pathways for the future

International Review of Social Psychology

Volumn 30, Issue 1, 2017, Pages 111-124

  Swiatkowski, Wojciech ^a   Dompnier, BenoÎt ^a  

^a UNIVERSITY OF LAUSANNE   (Switzerland)

Author keywords

Perspectivism; Publication standards; Questionable research practices; Replicability crisis; Social psychology; Statistical power

Indexed keywords

EID: 85019084066     PISSN: None     EISSN: 23978570     Source Type: Journal    
DOI: 10.5334/irsp.66     Document Type: Article

References (102)

1. Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543-554. DOI: https://doi.org/10.1177/1745691612459060.
2. Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666-678. DOI: https://doi.org/10.3758/s13428-011-0089-5.
3. Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230-244. DOI: https://doi.org/10.1037/0022-3514.71.2.230.
4. Baumeister, R. F., Bratslavsky, E., Muraven, M., & Tice, D. M. (1998). Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology, 74, 1252-1265. DOI: https://doi.org/10.1037/0022-3514.74.5.1252.
5. Bem, D. J. (2003). Writing the empirical journal article. In: Zanna, M. P., Darley, J. M., & Roediger, H. L., III. (Eds.), The compleat academic: A career guide (2nd ed., pp. 185-219). Washington, D.C.: American Psychological Association.
6. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. DOI: https://doi.org/10.1037/a0021524.
7. Branch, M. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24, 256-277. DOI: https://doi.org/10.1177/0959354314525282.
8. Brewer, M. B., & Crano, W. D. (2014). Research design and issues of validity. In: Reis, H. T., & Judd, C. M. (Eds.), Handbook of research methods in social and personality psychology (pp. 11-26), New York: Cambridge University Press.
9. Chalmers, A. F. (1982). What is this thing called science?: An assessment of the nature and status of science and its methods (2nd ed.). St. Lucia: University of Queensland Press.
10. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153. DOI: https://doi.org/10.1037/h0045186.
11. Cohen, J. (1965). Some statistical issues in psychological research. In: Wolman, B. B. (Ed.), Handbook of clinical psychology (pp. 95-121). New York: McGraw-Hill.
12. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312. DOI: https://doi.org/10.1037/0003-066X.45.12.1304.
13. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. DOI: https://doi.org/10.1037/0033-2909.112.1.155.
14. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003. DOI: https://doi.org/10.1037/0003-066X.49.12.997.
15. Cooper, M. L. (2016). Editorial. Journal of Personality and Social Psychology, 110, 431-434. DOI: https://doi.org/10.1037/pspp0000033.
16. Coulson, M., Healey, M., Fidler, F., & Cumming, G. (2010). Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Frontiers in Psychology, 1(26), 1-9. DOI: https://doi.org/10.3389/fpsyg.2010.00026.
17. Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science, 334, 1182. DOI: https://doi.org/10.1126/science.1216775.
18. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-Analysis. New York: Routledge.
19. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25, 7-29. DOIhttps://doi.org/10.1177/0956797613504966.
20. Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York: Palgrave Macmillan.
21. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290. DOI: https://doi.org/10.1177/1745691611406920.
22. Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 78, 78-89. DOI: https://doi.org/10.1016/j.jmp.2015.10.003.
23. Doyen, S., Klein, O., Pichon, C.-L., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLOS One, 7, e29081. DOI: https://doi.org/10.1371/journal.pone.0029081.
24. Etz, A., & Vandekerckhove, J. (2016). A Bayesian Perspective on the Reproducibility Project: Psychology. PLOS ONE, 11(2), e0149794. DOI: https://doi.org/10.1371/journal.pone.0149794.
25. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. DOI: https://doi.org/10.1007/s11192-011-0494-7.
26. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G∗Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191. DOI: https://doi.org/10.3758/BF03193146.
27. Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. Perspectives on Psychological Science, 7, 555-561. DOI: https://doi.org/10.1177/1745691612459059.
28. Fischhoff, B. (1975). Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty. Journal of Experimental Psychology: Human Perception and Performance, 1(3), 288-299. DOI: https://doi.org/10.1037/0096-1523.1.3.288.
29. Fiske, S. T. (2004). Mind the gap: In praise of informal sources of formal theory. Personality and Social Psychology Review, 8(2), 132-137. DOI: https://doi.org/10.1207/s15327957pspr0802-6.
30. Forstmeier, W., Wagenmakers, E.-J., & Parker, T. H. (2016). Detecting and avoiding likely false positive findings -A practical guide. Biological Reviews. DOI: https://doi.org/10.1111/brv.12315.
31. Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7(6), 585-594. DOI: https://doi.org/10.1177/1745691612459520.
32. Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past: Failures to replicate psi. Journal of Personality and Social Psychology, 103, 933-948. DOI: https://doi.org/10.1037/a0029709.
33. Gallistel, C. R. (2009). The importance of proving the null. Psychological Review, 116, 439-453. DOI: https://doi.org/10.1037/a0015251.
34. Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90-93. DOI: https://doi.org/10.1001/jama.295.1.90.
35. Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309-320. DOI: https://doi.org/10.1037/h0034436.
36. Gigerenzer, G. (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606. DOI: https://doi.org/10.1016/j.socec.2004.09.033.
37. Gigerenzer, G., Krauss, S., & Vitouch. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In: Kaplan, D. (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 391-408). Thousand Oaks, CA: Sage.
38. Gigerenzer, G., & Marewski, J. N. (2015). Surrogate science: The idol of a universal method for scientific inference. Journal of Management, 41, 421-440. DOI: https://doi.org/10.1177/0149206314547522.
39. Giner-Sorolla, R. (2012). Science or art?: How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562-571. DOI: https://doi.org/10.1177/1745691612457576.
40. Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Zwienenberg, M., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546-573. DOI: https://doi.org/10.1177/1745691616652873.
41. Hoekstra, R., Finch, S., Kiers, H. A., & Johnson, A. (2006). Probability as certainty: Dichotomous thinking and the misuse of p values. Psychonomic Bulletin & Review, 13, 1033-1037. DOI: https://doi.org/10.3758/BF03213921.
42. Hofstede, G. H., & Hofstede, G. (2001). Culture's consequences: Comparing values, behaviors, institutions and organizations across nations. Thousand Oaks, CA: Sage.
43. Inbar, Y. (2016). The association between "contextual dependence" and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences of the United States of America, 113(34), E4933-E4934. DOI: https://doi.org/10.1073/pnas.1608676113.
44. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. DOI: https://doi.org/10.1371/journal.pmed.0020124.
45. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. DOI: https://doi.org/10.1177/0956797611430953.
46. Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences, 110, 19313-19317. DOI: https://doi.org/10.1073/pnas.1313476110.
47. Jost, J. T., Banaji, M. R., & Prentice, D. A. (2004). Perspectivism in social psychology: The yin and yang of scientific progress. Washington, D.C.: American Psychological Association. DOI: https://doi.org/10.1037/10750-000.
48. Jost, J. T., & Kruglanski, A. W. (2002). The enstrangement of social constructionism and experimental social psychology: History of the rift and prospects of reconciliation. Personality and Social Psychology Review, 6(3), 168-187. DOI: https://doi.org/10.1207/S15327957PSPR0603-1.
49. Judd, C. M., & Gawronski, B. (2011). Editorial comment. Journal of Personality and Social Psychology, 100(3), 406. DOI: https://doi.org/10.1037/0022789.
50. Keppel, G. (1991). Design and analysis: A researcher's handbook (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
51. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. DOI: https://doi.org/10.1207/s15327957pspr0203-4.
52. Klein, O., Doyen, S., Leys, C., de Saldanha Gama, Miller, S., Questienne, L., & Cleeremens, A. (2012). Low hopes, high expectations: Expentancy effects and replicability of behavioral experiments. Perspectives on Psychological Science, 7(6), 572-584. DOI: https://doi.org/10.1177/1745691612463704.
53. Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
54. Lakatos, I. (1978). The methodology of scientific research programmes. London and New York: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511621123.
55. Lakens, D., & Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9, 278-292. DOI: https://doi.org/10.1177/1745691614528520.
56. Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584-585. DOI: https://doi.org/10.1126/science.aal3618.
57. Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537-542. DOI: https://doi.org/10.1177/1745691612460688.
58. Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 92, 224-253. DOI: https://doi.org/10.1037/0033-295X.98.2.224.
59. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163. DOI: https://doi.org/10.1037/1082-989X.9.2.147.
60. Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537-563. DOI: https://doi.org/10.1146/annurev.psych.59.103006.093735.
61. McGuire, W. J. (1983). A contextualist theory of knowledge: Its implications for innovation and reform in psychological research. In: Berkowitz, L. (Ed.), Advances in experimental social psychology, 16, 1-47. Orlando, FL: Academic Press. DOI: https://doi.org/10.1016/s0065-2601(08)60393-7.
62. McGuire, W. J. (1989). A perspectivist approach to the strategic planning of programmatic scientific research. In: Gholson, B., Shadish, W. R., Jr., Neimeyer, R. A., & Houts, A. C. (eds.), Psychology of science: Contribution to metascience (pp. 214-245). New York: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9781139173667.012.
63. McGuire, W. J. (1997). Creative hypothesis generating in psychology: Some useful heuristics. Annual Review of Psychology, 48, 1-30. DOIhttps://doi.org/10.1146/annurev.psych.48.1.1.
64. McGuire, W. J. (1999). Constructing social psychology: Creative and critical processes. Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511571206.
65. McGuire, W. J. (2004). A perspectivist approach to theory construction. Personality and Social Psychology Review, 8, 173-182. DOI: https://doi.org/10.1207/s15327957pspr0802-11.
66. Meehl, P. E. (1967). Theory-Testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115. DOI: https://doi.org/10.1086/288135.
67. Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834. DOI: https://doi.org/10.1037/0022-006X.46.4.806.
68. Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1, 108-141. DOI: https://doi.org/10.1207/s15327965pli0102-1.
69. Meehl, P. E. (1997). The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In: Harlow, L., Mulaik, S. A., & Steiger, J. H. (Eds.), What if there were no significance tests? (pp. 393-425). Mahwah, NJ: Erlbaum.
70. Mulder, J., & Wagenmakers, E.-J. (2016). Editors' introduction to the special issue "Bayes factors for testing hypotheses in psychological research: Practical relevance and new developments". Journal of Mathematical Psychology, 72, 1-5. DOIhttps://doi.org/10.1016/j.jmp.2016.01.002.
71. Neuliep, J. W., & Crandall, R. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8, 21-29.
72. Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought: Holistic versus analytic cognition. Psychological Review, 108, 291-310. DOI: https://doi.org/10.1037//0033-295X.108.2.291.
73. Norenzayan, A., & Heine, S. J. (2005). Psychological universals: What are they and how can we know? Psychological Bulletin, 131(5), 763-784. DOI: https://doi.org/10.1037/0033-2909.131.5.763.
74. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. DOI: https://doi.org/10.1177/1745691612459058.
75. Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2015). The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods, 48, 1205-1226. DOI: https://doi.org/10.3758/s13428-015-0664-2.
76. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660. DOI: https://doi.org/10.1177/1745691612462588.
77. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. DOI: https://doi.org/10.1126/science.aac4716.
78. Pashler, H., & Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530. DOI: https://doi.org/10.1177/1745691612465253.
79. Popper, K. R. (1959). The logic of scientific discovery. London: Hutchinson.
80. Richard, F. D., Bond, C. F. Jr., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363. DOI: https://doi.org/10.1037/1089-2680.7.4.331.
81. Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin, 113(3), 553-565. DOI: https://doi.org/10.1037/0033-2909.113.3.553.
82. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. DOI: https://doi.org/10.1037/0033-2909.86.3.638.
83. Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58, 646-656. DOI: https://doi.org/10.1037/0022-006X.58.5.646.
84. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-Tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237. DOI: https://doi.org/10.3758/PBR.16.2.225.
85. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551-566. DOIhttps://doi.org/10.1037/a0029487.
86. Schmidt, S. (2009). Shall we really do it again?: The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90-100. DOI: https://doi.org/10.1037/a0015108.
87. Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515-530. DOI: https://doi.org/10.1037/0022-3514.51.3.515.
88. Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316. DOI: https://doi.org/10.1037/0033-2909.105.2.309.
89. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. DOI: https://doi.org/10.1177/0956797611417632.
90. Stangor, C., & Lemay, E. P. Jr. (2016). Introduction to the special issue on methodological rigor and replicability. Journal of Experimental Social Psychology, 66, 1-3. DOI: https://doi.org/10.1016/j.jesp.2016.02.006.
91. Stukas, A. A., & Cumming, G. (2014). Interpreting effect sizes: Toward a quantitative cumulative social psychology. European Journal of Social Psychology, 44, 711-722. DOI: https://doi.org/10.1002/ejsp.2019.
92. Swiatkowski, W. (submitted). On the use of subjective probabilities in data analysis: An introduction to Bayesian statistics for social psychology starters.
93. Trafimow, D. (2009). The theory of reasoned action: A case study of falsification in psychology. Theory & Psychology, 19(4), 501-518. DOI: https://doi.org/10.1177/0959354309336319.
94. Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37(1), 1-2. DOI: https://doi.org/10.1080/01973533.2015.1012991.
95. Tressoldi, P. E. (2012). Replication unreliability in psychology: elusive phenomena or "elusive" statistical power? Frontiers in Psychology, 3, 1-5. DOI: https://doi.org/10.3389/fpsyg.2012.00218.
96. Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016a). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, 113, 6454-6459. DOI: https://doi.org/10.1073/pnas.1521897113.
97. Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016b). Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology. Proceedings of the National Academy of Sciences, 113, E4935-E4936. DOI: https://doi.org/10.1073/pnas.1609700113.
98. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804. DOI: https://doi.org/10.3758/BF03194105.
99. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432. DOI: https://doi.org/10.1037/a0022790.
100. Wallach, L., & Wallach, M. A. (2010). Some theories are unfalsifiable: A comment on Trafimow. Theory & Psychology, 20, 703-706. DOI: https://doi.org/10.1177/0959354310373676.
101. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70, 129-133. DOIhttps://doi.org/10.1080/00031305.2016.1154108.
102. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t-Tests. Perspectives on Psychological Science, 6(3), 291-298. DOI: https://doi.org/10.1177/1745691611406923.