Psychology, Science, and Knowledge Construction: Broadening Perspectives from the Replication Crisis

Annual Review of Psychology

Volumn 69, Issue , 2018, Pages 487-510

  Shrout, Patrick E ^a   Rodgers, Joseph L ^b  

^a NEW YORK UNIVERSITY   (United States)

^b VANDERBILT UNIVERSITY   (United States)

Author keywords

Methodology; Replication; Statistics

Indexed keywords

HUMAN; METHODOLOGY; PSYCHOLOGY; REPRODUCIBILITY; STATISTICS;

HUMANS; PSYCHOLOGY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; STATISTICS AS TOPIC;

EID: 85040343391     PISSN: 00664308     EISSN: 15452085     Source Type: Book Series    
DOI: 10.1146/annurev-psych-122216-011845     Document Type: Review

References (106)

1. Adolph KE, ColeWG, KomatiM, Garciaguirre JS, Badaly D, et al. 2012.Howdo you learn towalk?Thousands of steps and dozens of falls per day. Psychol. Sci. 23(11):1387-94
2. Aldrich J. 2008. R. A. Fisher on Bayes and Bayes' theorem. Bayesian Anal. 3(1):161-70
3. Alogna VK, Attaya MK, Aucoin P, Bahník S, Birch S, et al. 2014. Registered replication report. Perspect. Psychol. Sci. 9(5):556-78
4. Anderson SF, Maxwell SE. 2017. Addressing the "replication crisis": using original studies to design replication studies with appropriate statistical power. Multivar. Behav. Res. 52:305-24
5. Asendorpf JB, ConnerM, de Fruyt F, de Houwer J, Denissen JJA, et al. 2013. Recommendations for increasing replicability in psychology. Eur. J. Personal. 27(2):108-19
6. Bakker M, Wicherts JM. 2011. The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43(3):666-78
7. Baumeister. 2016. Charting the future of social psychology. J. Exp. Soc. Psychol. 66:153-58
8. Beasley WH, Rodgers JL. 2009. Resampling theory. In Handbook of Quantitative Methods in Psychology, ed. R Millsap, A Maydeu-Olivares, pp. 362-86. Thousand Oaks, CA: Sage
9. Bench SW, Rivera GN, Schlegel RJ, Hicks JA, Lench HC. 2017. Does expertise matter in replication? An examination of the reproducibility project: psychology. J. Exp. Soc. Psychol. 68:181-84
10. Benjamini Y, Hochberg Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B 57(1):289-300
11. Bhattacharjee Y. 2013. The mind of a con man. The New York Times Magazine, Apr. 28
12. Bonett DG. 2009. Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychol. Methods 14(3):225-38
13. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. 2009. Introduction to Meta-Analysis. Hoboken, NJ: Wiley
14. Box JF. 1978. R. A. Fisher: The Life of a Scientist. Hoboken, NJ: Wiley
15. Brandt MJ, IJzerman H, Dijksterhuis A, Farach FJ, Geller J, et al. 2014. The replication recipe: What makes for a convincing replication? J. Exp. Soc. Psychol. 50:217-24
16. Braver SL, Thoemmes FJ, Rosenthal R. 2014. Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9(3):333-42
17. Brown SD, Furrow D, Hill DF, Gable JC, Porter LP, Jacobs WJ. 2014. A duty to describe. Perspect. Psychol. Sci. 9(6):626-40
18. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, et al. 2013. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14(5):365-76
19. Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences. Abingdon, UK: Routledge. 2nd ed.
20. Cohen J. 1994. The Earth is round (p 0.05). Am. Psychol. 49(12):997-1003
21. Coyne JC. 2016. Replication initiatives will not salvage the trustworthiness of psychology.BMCPsychol. 4(1):28
22. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. 1972. The Dependability of Behavioral Measurements. Hoboken, NJ: Wiley
23. Cumming G. 2014. The new statistics. Psychol. Sci. 25(1):7-29
24. Curran PJ, Hussong AM. 2009. Integrative data analysis. Psychol. Methods 14(2):81-100
25. De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. 2005. Is this clinical trial fully registered? A statement from the International Committee ofMedical Journal Editors. N. Engl. J. Med. 352(23):2436-38
26. DickersinK, RennieD. 2012. The evolution of trial registries and their use to assess the clinical trial enterprise. JAMA 307(17):1861-64
27. Ding N, Melloni L, Zhang H, Tian X, Poeppel D. 2016. Cortical tracking of hierarchical linguistic structures in connected speech. Nat. Neurosci. 19(1):158-64
28. Ebersole CR, Atherton OE, Belanger AL, Skulborstad HM, Allen JM, et al. 2016. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67:68-82
29. Edwards J, Berry J. 2010. The presence of something or the absence of nothing: increasing theoretical precision in management research. Organ. Res. Methods 13(4):668-89
30. Efron B. 1979. Bootstrap methods: another look at the jackknife. Ann. Stat. 7(1):1-26
31. Efron B, Gong G. 1983. A leisurely look at the bootstrap, jackknife, and cross-validation. Am. Stat. 37:36-48
32. Fabrigar LR, Wegener DT. 2016. Conceptualizing and evaluating the replication of research results. J. Exp. Soc. Psychol. 66:68-80
33. Fiedler K. 2017. What constitutes strong psychological science? The (neglected) role of diagnosticity and a priori theorizing. Perspect. Psychol. Sci. 12(1):46-61
34. Finkel EJ, Eastwick PW, Reis HT. 2015. Best research practices in psychology: illustrating epistemological and pragmatic considerations with the case of relationship science. J. Personal. Soc. Psychol. 108(2):275-97
35. Finkel EJ, Eastwick PW, Reis HT. 2017. Replicability and other features of a high-quality science: toward a balanced and empirical approach. J. Personal. Soc. Psychol. 113(2):244-53
36. Fisher RA. 1925. Statistical Methods for Research Workers. Edinburgh, Scotl.: Oliver &Boyd
37. Fisher RA. 1958. The nature of probability. Centen. Rev. Arts Sci. 2:261-74
38. Francis G. 2012a. The psychology of replication and replication in psychology. Perspect. Psychol. Sci. 7(6):585-94
39. Francis G. 2012b. Publication bias and the failure of replication in experimental psychology. Psychon. Bull. Rev. 19(6):975-91
40. Francis G. 2012c. Too good to be true: publication bias in two prominent studies from experimental psychology. Psychon. Bull. Rev. 19(2):151-56
41. Gallistel CR. 2009. The importance of proving the null. Psychol. Rev. 116(2):439-53
42. Giner-Sorolla R. 2016. Approaching a fair deal for significance and other concerns. J. Exp. Soc. Psychol. 65:1-6
43. Greenwald AG. 1976. An editorial. J. Personal. Soc. Psychol. 33(1):1-7
44. Hagger MS, Chatzisarantis NLD. 2016. A multilab preregistered replication of the ego-depletion effect. Perspect. Psychol. Sci. 11(4):546-73
45. Ioannidis JPA. 2005. Why most published research findings are false. PLOS Med. 2(8):e124
46. Kawakami K. 2015. Editorial. J. Personal. Soc. Psychol. 108(1):58-59
47. Kerr N. 1998. HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2(3):196-217
48. Kidwell MC, Lazarević LB, Baranski E, HardwickeTE, Piechowski S, et al. 2016. Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLOS Biol. 14(5):e1002456
49. Klein RA, Ratliff KA, Vianello M, Adams RB, Bahnik S, et al. 2014. Investigating variation in replicability: a "many labs" replication project. Soc. Psychol. 45(3):142-52
50. Kruschke JK. 2015. Doing Bayesian Data Analysis: A Tutorial with R, JAGS and Stan. Burlington, MA: Academic. 2nd ed.
51. Kruschke JK, Liddell TM. 2017. The Bayesian new statistics: hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychon. Bull. Rev. 24:1-29
52. LeBel EP, Campbell L, Loving TJ. 2017. Benefits of open and high-powered research outweigh costs. J. Personal. Soc. Psychol. 113(2):254-61
53. Ledgerwood A, Sherman JW. 2012. Short, sweet, and problematic? The rise of the short report in psychological science. Perspect. Psychol. Sci. 7(1):60-66
54. Ledgerwood A, Soderberg C, Sparks J. 2017. Designing a study to maximize informational value. In Toward a More Perfect Psychology: Improving Trust, Accuracy, and Transparency, ed.MCMakel, JA Plucker, pp. 33-58. Washington, DC: Am. Psychol. Assoc.
55. Lindsay DS. 2015. Replication in psychological science. Psychol. Sci. 26(12):1827-32
56. Maxwell SE, LauMY, Howard GS. 2015. Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? Am. Psychol. 70(6):487-98
57. McArdle JJ, Ritschard G. 2014. Contemporary Issues in Exploratory Data Mining in the Behavioral Sciences. Abingdon, UK: Routledge
58. McShane B, Böckenholt U. 2014. You cannot step into the same river twice:when power analyses are optimistic. Perspect. Psychol. Sci. 9(6):612-25
59. McShane BB, Böckenholt U, Hansen KT. 2016. Adjusting for publication bias in meta-analysis. Perspect. Psychol. Sci. 11(5):730-49
60. Meiser T. 2011. Much pain, little gain? Paradigm-specific models and methods in experimental psychology. Perspect. Psychol. Sci. 6(2):183-91
61. Mill JS. 2008 (1843). A System of Logic. London: Longmans, Green
62. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, et al. 2017. A manifesto for reproducible science. Nat. Hum. Behav. 1(1):21
63. Nelson LD, Simmons J, Simonsohn U. 2018. Psychology's renaissance. Annu. Rev. Psychol. 69. In press
64. Neyman J, Pearson ES. 1928. On the use and interpretation of certain test criteria for purposes of statistical inference: part I. Biometrika 20A(1/2):175-240
65. Neyman J, Pearson ES. 1933. On the problem of themost efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Math. Phys. Eng. Sci. 231:289-337
66. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, et al. 2015. Promoting an open research culture. Science 348(6242):1422-25
67. NosekBA, Bar-AnanY. 2012. Scientific utopia: I. Opening scientific communication. Psychol. Inq. 23(3):217-43
68. Nosek BA, Lakens DD. 2014. Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45(3):137-41
69. Nosek BA, Spies J, Motyl M. 2012. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7(6):615-31
70. Offord DR, Kraemer HC, Kazdin AE, Jensen PS, Harrington R. 1998. Lowering the burden of suffering from child psychiatric disorder: trade-offs among clinical, targeted, and universal interventions. J. Am. Acad. Child Adolesc. Psychiatry 37(7):686-94
71. Open Sci. Collab. 2015. Estimating the reproducibility of psychological science. Science 349(6251):aac4716
72. Open Sci. Collab. 2017. Maximizing the reproducibility of your research. In Psychological ScienceUnder Scrutiny: Recent Challenges and Proposed Solutions, ed. SO Lilienfeld, ID Waldman, pp. 3-21. New York: Wiley
73. Overall NC, Girme YU, Lemay J, Edward P, Hammond MD. 2014. Attachment anxiety and reactions to relationship threat: the benefits and costs of inducing guilt in romantic partners. J. Personal. Soc. Psychol. 106(2):235-56
74. Overath T, McDermott JH, Zarate JM, Poeppel D. 2015. The cortical analysis of speech-specific temporal structure revealed by responses to sound quilts. Nat. Neurosci. 18(6):903-11
75. OzomaroU, WahlestedtC, NemeroffCB. 2013. Personalizedmedicine in psychiatry: problems and promises. BMC Med. 11(1):132
76. Paolacci G, Chandler J. 2014. Inside the Turk: understanding Mechanical Turk as a participant pool. Curr. Dir. Psychol. Sci. 23(3):184-88
77. Perugini M, Gallucci M, Costantini G. 2014. Safeguard power as a protection against imprecise power estimates. Perspect. Psychol. Sci. 9(3):319-32
78. Richard FD, Bond CF, Stokes-Zoota JJ. 2003. One hundred years of social psychology quantitatively described. Rev. Gen. Psychol. 7(4):331-63
79. Rogers JL, Howard KI, Vessey JT. 1993. Using significance tests to evaluate equivalence between two experimental groups. Psychol. Bull. 113(3):553-65
80. Rossi JS. 1990. Statistical power of psychological research. J. Consult. Clin. Psychol. 58(5):646-56
81. Scheibehenne B, Jamil T, Wagenmakers E. 2016. Bayesian evidence synthesis can reconcile seemingly inconsistent results. Psychol. Sci. 27(7):1043-46
82. Schimmack U. 2012. The ironic effect of significant results on the credibility ofmultiple-study articles. Psychol. Methods 17(4):551-66
83. Schmidt FL, Oh I. 2016. The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Arch. Sci. Psychol. 4(1):32-37
84. Schooler JW, Engstler-Schooler TY. 1990. Verbal overshadowing of visual memories: Some things are better left unsaid. Cogn. Psychol. 22(1):36-71
85. Schweinsberg M, Madan N, Vianello M, Sommer SA, Jordan J, et al. 2016. The pipeline project: prepublication independent replications of a single laboratory's research pipeline. J. Exp. Soc. Psychol. 66:55-67
86. Seaman MA, Serlin RC. 1998. Equivalence confidence intervals for two-group comparisons of means. Psychol. Methods 3(4):403-11
87. Shrout PE, Yip-Bannicq M. 2017. Inferences about competing measures based on patterns of binary significance tests are questionable. Psychol. Methods 22(1):84-93
88. Simmons J, Nelson L, SimonsohnU. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22(11):1359-66
89. Simons DJ, Holcombe AO, Spellman BA. 2014. An introduction to registered replication reports at Perspectives on Psychological Science. Perspect. Psychol. Sci. 9(5):552-55
90. Simonsohn U. 2013. Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychol. Sci. 24(10):1875-88
91. Sripada C, Kessler D, Jonides J. 2014. Methylphenidate blocks effort-induced depletion of regulatory control in healthy volunteers. Psychol. Sci. 25(6):1227-34
92. Stewart LA, Parmar MKB. 1993. Meta-analysis of the literature or of individual patient data: Is there a difference? Lancet 341(8842):418-22
93. Stigler SM. 1986. TheHistory of Statistics: The Measurement of Uncertainty Before 1900. Cambridge, MA: Harvard Univ. Press
94. StroebeW. 2016. Are most published social psychological findings false? J. Exp. Soc. Psychol. 66:134-44
95. Tackett JL, Lilienfeld SO, Johnson SL, Krueger RF, Miller JD, et al. 2017. It's time to broaden the replicability conversation: thoughts for and from clinical psychological science. Perspect. Psychol. Sci. 12(5):742-56
96. Taylor DJ, Muller KE. 1996. Bias in linear model power and sample size calculation due to estimating noncentrality. Commun. Stat. Theory Methods 25(7):1595-610
97. Tukey JW. 1977. Exploratory Data Analysis. Reading, MA: Addison-Wesley
98. Van Bavel JJ, Mende-Siedlecki P, Brady WJ, Reinero DA. 2016. Contextual sensitivity in scientific reproducibility. PNAS 113(23):6454-59
99. Vazire S. 2016. Editorial. Soc. Psychol. Personal. Sci. 7(1):3-7
100. Verhagen J, Wagenmakers E. 2014. Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen. 143(4):1457-75
101. Vul E, Harris C, Winkielman P, Pashler H. 2009. Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4(3):274-90
102. Wade N. 2010. Inquiry on Harvard lab threatens ripple effect. The New York Times, Aug. 12
103. Wagenmakers E, Verhagen J, Ly A. 2016. How to quantify the evidence for the absence of a correlation. Behav. Res. Methods 48(2):413-26
104. Wagenmakers E, Wetzels R, Borsboom D, van der Maas HLJ, Kievit RA. 2012. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7(6):632-38
105. Westfall J, Judd CM, Kenny DA. 2015. Replicating studies in which samples of participants respond to samples of stimuli. Perspect. Psychol. Sci. 10(3):390-99
106. Zabell S. 1989. R. A. Fisher on the history of inverse probability. Stat. Sci. 4(3):247-56