Is psychology suffering from a replication crisis?: What does 'failure to replicate' really mean?

American Psychologist

Volumn 70, Issue 6, 2015, Pages 487-498

  Maxwell, Scott E ^a   Lau, Michael Y ^b   Howard, George S ^a  

^a UNIVERSITY OF NOTRE DAME   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

Bayesian methods; Equivalence tests; False positive results; Metaanalysis; Statistical power

Indexed keywords

BAYES THEOREM; HUMAN; METHODOLOGY; PROCEDURES; PSYCHOLOGY; REPRODUCIBILITY; SAMPLE SIZE;

BAYES THEOREM; HUMANS; PSYCHOLOGY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE;

EID: 84941060169     PISSN: 0003066X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0039400     Document Type: Article

References (73)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, Š., Birch, S., Birt, A. R., . . . Zwaan, R. A. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556-578. http://dx.doi.org/10.1177/1745691614545653
2. Bargh, J. A., & Shalev, I. (2012). The substitutability of physical and social warmth in daily life. Emotion, 12, 154-162. http://dx.doi.org/ 10.1037/a0023527
3. Bartlett, T. (2012). Is psychology about to come undone? The Chronicle of Higher Education. Retrieved November 23 2014, from http:// chronicle.com/blogs/percolator/is-psychology-about-to-come-undone/ 29045
4. Bartlett, T. (2014). Replication crisis in psychology research turns ugly and odd. The Chronicle of Higher Education. Retrieved November 23 2014, from http://chronicle.com/article/Replication-Crisis-in/147301/
5. Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483, 531-533. http://dx.doi.org/ 10.1038/483531a
6. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. http://dx.doi.org/10.1037/ a0021524
7. Bonett, D. G. (2012). Replication-extension studies. Current Directions in Psychological Science, 21, 409-412. http://dx.doi.org/10.1177/ 0963721412459512
8. Brandt, M. J., IJzerman, H., Dijkstehuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., . . . van't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224. http://dx.doi.org/10.1016/j.jesp.2013 .10.005
9. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365-376. http://dx.doi.org/10.1038/nrn3475
10. Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9, 40-48. http://dx.doi.org/10.1177/ 1745691613513470
11. Chow, S.-C., Shao, J., & Wang, H. (2003). Sample size calculations in clinical research. Boca Raton, FL: Taylor and Francis.
12. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
13. Cook, T. D., Gruder, C. L., Henningan, K. M., & Faly, B. R. (1979). History of the sleeper effect: Some logical pitfalls in accepting the null hypothesis. Psychological Bulletin, 86, 662-679. http://dx.doi.org/ 10.1037/0033-2909.86.4.662
14. Dallow, N., & Fina, P. (2011). The perils with the misuse of predictive power. Pharmaceutical Statistics, 10, 311-317. http://dx.doi.org/ 10.1002/pst.467
15. Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS ONE, 7, e29081. http://dx.doi.org/10.1371/journal.pone.0029081
16. Finch, S., Cumming, G., & Thomason, N. (2001). Reporting of statistical inference in the Journal of Applied Psychology: Little evidence of reform. Educational and Psychological Measurement, 61, 181-210. http://dx.doi.org/10.1177/00131640121971167
17. Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review, 18, 3-12. http://dx.doi.org/10.1177/1088868313507536
18. Galak, J., Leboeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past: Failures to replicate _. Journal of Personality and Social Psychology, 103, 933-948. http://dx.doi.org/10.1037/a0029709
19. Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (Sign) and Type M (Magnitude) errors. Perspectives on Psychological Science, 9, 641-651. http://dx.doi.org/10.1177/ 1745691614551642
20. Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102, 460-465. http://dx.doi.org/10.1511/2014 .111.460
21. Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60, 328-331. http://dx.doi.org/10.1198/000313006X152649
22. Gonzalez, R. (2009). Data analysis for experimental design. New York, NY: Guilford Press.
23. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20. http://dx.doi.org/10.1037/ h0076157
24. Greenwald, A. G., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996). Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology, 33, 175-183. http://dx.doi.org/10.1111/ j.1469-8986.1996.tb02121.x
25. Hedges, L. V. (1987). How hard is hard science, how soft is soft science? The empirical cumulativeness of research. American Psychologist, 42, 443-455. http://dx.doi.org/10.1037/0003-066X.42.5.443
26. Hedges, L. V. (1998). Bayesian meta-analysis. In B. S. Everitt & G. Dunn (Eds.), Statistical analysis of medical data: New developments (pp. 251-275) New York, NY: Oxford University Press.
27. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504. http://dx.doi.org/ 10.1037/1082-989X.3.4.486
28. Henrion, M., & Fischoff, B. (1986). Assessing uncertainty in physical constants. American Journal of Physics, 54, 791-798. http://dx.doi.org/ 10.1119/1.14447
29. Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009). A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society Series A, Statistics in Society, 172, 137-159. http:// dx.doi.org/10.1111/j.1467-985X.2008.00552.x
30. Hunter, J. E. (2001). The desperate need for replications. Journal of Consumer Research, 28, 149-158. http://dx.doi.org/10.1086/321953
31. Ioannidis, J. P. A. (2005). Why most published research findings are false.PLoSMedicine, 2, e124. http://dx.doi.org/10.1371/journal.pmed .0020124
32. Jeffreys, H. (1961). The theory of probability (3rd ed.). New York, NY: Oxford University Press.
33. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524-532. http://dx.doi.org/10.1177/ 0956797611430953
34. Kahneman, D. (2012, September 26). A proposal to deal with questions about priming effects. [Letter emailed to social priming researchers]. Retrieved from http://www.nature.com/polopoly_fs/ 7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf
35. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795. http://dx.doi.org/ 10.1080/01621459.1995.10476572
36. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspectives on Psychological Science, 6, 299-312. http://dx.doi.org/10.1177/1745691611406925
37. Kruschke, J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: Generalv, 142, 573-603. http://dx.doi.org/ 10.1037/a0029146
38. Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial with R, JAGS, and BUGS (2nd ed.). Burlington, MA: Elsevier.
39. Kuiper, R. M., Buskens, V., Raub, W., & Hoijtink, H. (2013). Combining statistical evidence from several studies: A method using Bayesian updating and an example from research on trust problems in social and economic exchange. Sociological Methods & Research, 42, 60-81. http://dx.doi.org/10.1177/0049124112464867
40. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112. http://dx.doi.org/10.1111/j.2044-8317.1978.tb00578.x
41. Lau, J., Antman, E. M., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. The New England Journal of Medicine, 327, 248-254. http://dx.doi.org/10.1056/NEJM199207233270406
42. Masson, M. E. J. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. Behavior Research Methods, 43, 679-690. http://dx.doi.org/10.3758/s13428-010-0049-5
43. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163. http://dx.doi.org/10.1037/1082-989X.9.2.147
44. McShane, B. B., & Böckenholt, U. (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives on Psychological Science, 9, 612-625. http://dx.doi.org/10.1177/ 1745691614548513
45. Moonesinghe, R., Khoury, M. J., & Janssens, A. C. (2007). Most published research findings are false-but a little replication goes a long way. PLoS Medicine, 4, e28. http://dx.doi.org/10.1371/journal.pmed .0040028
46. Nosek, B. A., & Lakens, D. (2014). A method to increase the credibility of published results. Social Psychology, 45, 137-141. http://dx.doi.org/ 10.1027/1864-9335/a000192
47. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. http://dx .doi.org/10.1177/1745691612459058
48. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657-660. http://dx.doi.org/10.1177/ 1745691612462588
49. Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PLoS ONE, 7, e42510. http://dx.doi.org/10.1371/journal.pone.0042510
50. Pashler, H., & Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530. http:// dx.doi.org/10.1177/1745691612465253
51. Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem's "retroactive facilitation of recall" effect. PLoS ONE, 7, e33423. http://dx.doi.org/10.1371/journal .pone.0033423
52. Roediger, H. L. (2012, February). Psychology's woes and a partial cure: The value of replication. Observer,. 25 Retrieved from http://www .psychologicalscience.org/index.php/publications/observer/2012/ february-12/psychologys-woes-and-a-partial-cure-the-value-ofreplication. html
53. Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin, 113, 553-565. http://dx.doi.org/10.1037/0033-2909 .113.3.553
54. Rosenthal, R., & Rubin, D. B. (1994). The counternull value of an effect size: A new statistic. Psychological Science, 5, 329-334. http://dx.doi .org/10.1111/j.1467-9280.1994.tb00281.x
55. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin&Review, 16, 225-237. http://dx.doi.org/10.3758/ PBR.16.2.225
56. Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173-1181. http://dx.doi.org/10.1037/0003-066X.47 .10.1173
57. Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36-71. http://dx.doi.org/10.1016/0010-0285(90)90003-M
58. Seaman, M. A., & Serlin, R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, 3, 403-411. http://dx.doi.org/10.1037/1082-989X.3.4.403
59. Serlin, R. C., & Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73-83. http://dx.doi.org/10.1037/0003-066X.40.1.73
60. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth.
61. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. http://dx.doi.org/10.1177/0956797611417632
62. Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76-80. http://dx.doi.org/10.1177/1745691613514755
63. Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014). An introduction to Registered Replication Reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9, 552-555. http://dx .doi.org/10.1177/1745691614543974
64. Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59-71. http://dx.doi.org/10.1177/1745691613514450
65. Taylor, D. J., & Muller, K. E. (1996). Bias in linear model power and sample size calculation due to estimating noncentrality. Communications in Statistics Theory and Methods, 25, 1595-1610. http://dx.doi .org/10.1080/03610929608831787
66. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. http://dx.doi.org/10.1037/h0031322
67. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804. http://dx.doi.org/10.3758/BF03194105
68. Wagenmakers, E-J., Lee, M. D., Lodewyckx, T., & Iverson, G. (2008). Bayesian versus frequentist inference. In H. Hoijtink, I. Klugkist, & P. A. Goelen (Eds.), Bayesian Evaluation of Informative Hypotheses (pp. 181-207). New York, NY: Springer. http://dx.doi.org/10.1007/ 978-0-387-09612-4_9
69. Williams, L. E., & Bargh, J. A. (2008). Keeping one's distance: The influence of spatial distance cues on affect and evaluation. Psychological Science, 19, 302-308. http://dx.doi.org/10.1111/j.1467-9280.2008.02084.x
70. Wilson, D. B., & Shadish, W. R. (2006). On blowing trumpets to the tulips: To prove or not to prove the null hypothesis-Comment on Bösch, Steinkamp, and Boller (2006). Psychological Bulletin, 132, 524-528. http://dx.doi.org/10.1037/0033-2909.132.4.524
71. Wortman, J., Donnellan, M. B., & Lucas, R. E. (2014). Can physical warmth (or coldness) predict trait loneliness? A replication of Bargh and Shalev (2012). Archives of Scientific Psychology, 2, 13-19. http:// dx.doi.org/10.1037/arc0000007
72. Yuan, K.-H., & Maxwell, S. E. (2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30, 141-167. http://dx.doi.org/10.3102/10769986030002141
73. Zellner, A., & Siow, A. (1980). Posterior odds ratios for selected regression hypotheses. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian statistics: Proceedings of the first international meeting (pp. 585-603). Valencia, Spain: University of Valencia Press.