Contextual sensitivity in scientific reproducibility

Proceedings of the National Academy of Sciences of the United States of America

Volumn 113, Issue 23, 2016, Pages 6454-6459

  Van Bavel, Jay J ^a   Mende Siedlecki, Peter ^a   Brady, William J ^a   Reinero, Diego A ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Context; Meta science; Psychology; Replication; Reproducibility

Indexed keywords

ARTICLE; BEHAVIOR; PRIORITY JOURNAL; PSYCHOLOGIST; REPLICATION STUDY; REPRODUCIBILITY; RESEARCH; HUMAN; PSYCHOLOGY; PUBLISHING; SCIENCE; STATISTICS AND NUMERICAL DATA;

HUMANS; PSYCHOLOGY; PUBLISHING; REPRODUCIBILITY OF RESULTS; RESEARCH; SCIENCE;

EID: 84973316189     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1521897113     Document Type: Article

References (92)

1. Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K (2002) A comprehensive review of genetic association studies. Genet Med 4(2):45-61.
2. Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10(9):712.
3. Begley CG, Ellis LM (2012) Drug development: Raise standards for preclinical cancer research. Nature 483(7391):531-533.
4. Reaves ML, Sinha S, Rabinowitz JD, Kruglyak L, Redfield RJ (2012) Absence of detectable arsenate in DNA from arsenate-grown GFAJ-1 cells. Science 337(6093):470-473.
5. Chang AC, Li P (2015) Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say "Usually Not", Finance and Economics Discussion Series 2015-083. (Board of Governors of the Federal Reserve System, Washington, DC).
6. Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2(8):e124.
7. Kaplan RM, Irvin VL (2015) Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One 10(8):e0132382.
8. Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility in preclinical research. PLoS Biol 13(6):e1002165.
9. Bakker M, Wicherts JM (2011) The (mis)reporting of statistical results in psychology journals. Behav Res Methods 43(3):666-678.
10. Fiedler K (2011) Voodoo correlations are everywhere - not only in neuroscience. Perspect Psychol Sci 6(2):163-171.
11. García-Pérez MA (2012) Statistical conclusion validity: Some common threats and simple remedies. Front Psychol 3:325.
12. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23(5):524-532.
13. Pashler H, Wagenmakers EJ (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci 7(6): 528-530.
14. Spellman BA (2012b) Introduction to the special section on research practices. Perspect Psychol Sci 7(6):655-656.
15. Bollen K, Cacioppo JT, Kaplan RM, Krosnick JA, Olds JL (2015) Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science: Report of the Subcommittee on Replicability in Science, Advisory Committee to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences. Available at www.nsf.gov/sbe/AC-Materials/SBE-Robust-and-Reliable-Research-Report.pdf. Accessed May 10, 2016.
16. Open Science Collaboration (2015) Psychology. Estimating the reproducibility of psychological science. Science 349(6251):aac4716.
17. Sample I (2015) Study delivers bleak verdict on validity of psychology experiment results. Available at https://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdicton- validity-of-psychology-experiment-results. Accessed May 10, 2016.
18. Efferson ADP (2015) How many laws are based on psychology's bad science. Available at thefederalist.com/2015/09/08/how-many-laws-are-based-on-psychologys-bad-science/. Accessed May 10, 2016.
19. Ferguson CJ (2015) "Everybody knows psychology is not a real science": Public perceptions of psychology and how we can improve our relationship with policymakers, the scientific community, and the general public. Am Psychol 70(6):527-542.
20. Lilienfeld SO (2012) Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. Am Psychol 67(2):111-129.
21. Ferguson CJ, Heene M (2012) A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspect Psychol Sci 7(6):555-561.
22. Rosenthal R (1979) The file drawer problem and tolerance for null results. Psychol Bull 86(3):638-641.
23. Cohen J (1962) The statistical power of abnormal-social psychological research: A review. J Abnorm Soc Psychol 65:145-153.
24. Maxwell SE, Lau MY, Howard GS (2015) Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? Am Psychol 70(6):487-498.
25. Vankov I, Bowers J, Munafò MR (2014) On the persistence of low power in psychological science. Q J Exp Psychol (Hove) 67(5):1037-1040.
26. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22(11):1359-1366.
27. Kerr NL (1998) HARKing: Hypothesizing after the results are known. Pers Soc Psychol Rev 2(3):196-217.
28. Yong E (2012) Replication studies: Bad copy. Nature 485(7398):298-300.
29. Young NS, Ioannidis JPA, Al-Ubaydli O (2008) Why current publication practices may distort science. PLoS Med 5(10):e201.
30. Cumming G (2014) The new statistics: Why and how. Psychol Sci 25(1):7-29.
31. Lindsay DS (2015) Replication in psychological science. Psychol Sci 26(12):1827-1832.
32. Feldman-Barrett L (2015) Psychology is not in crisis. Available at www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html?-r=1. Accessed May 10, 2016.
33. Gelman A (2014) The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. J Manage 41(2):632-643.
34. McGuire WJ (2013) An additional future for psychological science. Perspect Psychol Sci 8(4):414-423.
35. Klein O, et al. (2012) Low hopes, high expectations expectancy effects and the replicability of behavioral experiments. Perspect Psychol Sci 7(6):572-584.
36. Stroebe W, Strack F (2014) The alleged crisis and the illusion of exact replication. Perspect Psychol Sci 9(1):59-71.
37. Touhey JC (1981) Replication failures in personality and social psychology negative findings or mistaken assumptions? Pers Soc Psychol Bull 7(4):593-595.
38. Lewin K (1936) Principles of Topological Psychology (McGraw-Hill, New York) trans Heider F and Heider G.
39. Mischel W (1977) The interaction of person and situation. Personality at the Crossroads: Current Issues in Interactional Psychology, eds Magnnusson D, Endler D (Lawrence Erlbaum Associates, Hillsdale, NJ), pp 333-352.
40. Rousseau DM, Fried Y (2001) Location, location, location: Contextualizing organizational research. J Organ Behav 22(1):1-13.
41. Sarason IG, Smith RE, Diener E (1975) Personality research: Components of variance attributable to the person and the situation. J Pers Soc Psychol 32(2):199-204.
42. Weick KE (1996) Enactment and the boundaryless career. The Boundaryless Career: A New Employment Principle for a New Organizational Era, eds Arthur MB, Rousseau DM (Oxford Univ Press, New York), pp 40-57.
43. Camerer CF, Loewenstein G, Rabin M, eds (2011) Advances in Behavioral Economics (Princeton Univ Press, Princeton, NJ).
44. Fiske ST, Gilbert DT, Lindzey G (2010) Handbook of Social Psychology (John Wiley & Sons, Hoboken, NJ).
45. Goodin RE, ed (2009) The Oxford Handbook of Political Science (Oxford Univ Press, New York).
46. Hedström P, Bearman P, eds (2009) The Oxford Handbook of Analytical Sociology (Oxford Univ Press, New York).
47. Anderson CJ, et al. (2016) Response to comment on "Estimating the reproducibility of psychological science". Science 351(6277):1037.
48. Kahneman D (2014) A new etiquette for replication. Soc Psychol 45(4):299-311.
49. Reardon S (2016) A mouse's house may ruin experiments. Available at www.nature. com/news/a-mouse-s-house-may-ruin-experiments-1.19335. Accessed May 10, 2016.
50. Schaffer S (1989) Glass works: Newton's prisms and the uses of experiment. The Uses of Experiment: Studies in the Natural Sciences, eds Gooding D, Pinch T, Schaffer S (Cambridge Univ Press, Cambridge, UK), pp 67-104.
51. Bloom P (2016) Psychology's Replication Crisis Has a Silver Lining. Available at www. theatlantic.com/science/archive/2016/02/psychology-studies-replicate/468537/. Accessed May 10, 2016.
52. Roberts B (2015) The new rules of research. Available at https://pigee.wordpress.com/2015/09/17/the-new-rules-of-research/. Accessed May 10, 2016.
53. Srivastava S (2015) Moderator interpretations of the Reproducibility Project. Available at https://hardsci.wordpress.com/2015/09/02/moderator-interpretations-of-thereproducibility- project/. Accessed May 10, 2016.
54. Koole SL, Lakens D (2012) Rewarding replications a sure and simple way to improve psychological science. Perspect Psychol Sci 7(6):608-614.
55. Simons DJ (2014) The value of direct replication. Perspect Psychol Sci 9(1):76-80.
56. Gilbert DT, King G, Pettigrew S, Wilson TD (2016) Comment on "Estimating the reproducibility of psychological science". Science 351(6277):1037.
57. Gilbert DT, King G, Pettigrew S, Wilson TD (2016) More on "Estimating the Reproducibility of Psychological Science". Available at projects.iq.harvard.edu/files/psychology-replications/files/gkpw-post-publication-response.pdf?platform=hootsuite. Accessed May 10, 2016.
58. Lakens D (2016) The statistical conclusions in Gilbert et al (2016) are completely invalid. Available at daniellakens.blogspot.nl/2016/03/the-statistical-conclusions-ingilbert. html. Accessed May 10, 2016.
59. Srivastava S (2016) Evaluating a new critique of the Reproducibility Project. Available at https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibilityproject/. Accessed May 10, 2016.
60. Etz A, Vandekerckhove J (2016) A Bayesian perspective on the reproducibility project: Psychology. PLoS One 11(2):e0149794.
61. Klein RA, et al. (2014) Investigating variation in replicability. Soc Psychol 45(3): 142-152.
62. Schweinsberg M, et al. (2016) The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. J Exp Soc Psychol, in press.
63. Eyal T, Liberman N, Trope Y (2008) Judging near and distant virtue and vice. J Exp Soc Psychol 44(4):1204-1209.
64. Schooler JW, Engstler-Schooler TY (1990) Verbal overshadowing of visual memories: Some things are better left unsaid. Cognit Psychol 22(1):36-71.
65. Shih M, Pittinsky TL, Ambady N (1999) Stereotype susceptibility: Identity salience and shifts in quantitative performance. Psychol Sci 10(1):80-83.
66. Williams LE, Bargh JA (2008) Experiencing physical warmth promotes interpersonal warmth. Science 322(5901):606-607.
67. Shrout PE, Fleiss JL (1979) Intraclass correlations: Uses in assessing rater reliability. Psychol Bull 86(2):420-428.
68. Turk-Browne NB, Isola PJ, Scholl BJ, Treat TA (2008) Multidimensional visual statistical learning. J Exp Psychol Learn Mem Cogn 34(2):399-407.
69. Harmon-Jones E, Harmon-Jones C, Fearn M, Sigelman JD, Johnson P (2008) Left frontal cortical activation and spreading of alternatives: Tests of the action-based model of dissonance. J Pers Soc Psychol 94(1):1-15.
70. Colzato LS, et al. (2008) How does bilingualism improve executive control? A comparison of active and reactive inhibition mechanisms. J Exp Psychol Learn Mem Cogn 34(2):302-312.
71. Fiedler K (2008) The ultimate sampling dilemma in experience-based decision making. J Exp Psychol Learn Mem Cogn 34(1):186-203.
72. Purdie-Vaughns V, Steele CM, Davies PG, Ditlmann R, Crosby JR (2008) Social identity contingencies: How diversity cues signal threat or safety for African Americans in mainstream institutions. J Pers Soc Psychol 94(4):615-630.
73. Richard FD, Bond CF, Jr, Stokes-Zoota JJ (2003) One hundred years of social psychology quantitatively described. Rev Gen Psychol 7(4):331-363.
74. Bohannon J (2015) Reproducibility. Many psychology papers fail replication test. Science 349(6251):910-911.
75. Baker M (August 27th, 2015) Over half of psychology studies fail reproducibility test. Available at www.nature.com/news/over-half-of-psychology-studies-fail-reproducibilitytest- 1.18248. Accessed May 10, 2016.
76. Brandt MJ, et al. (2014) The replication recipe: What makes for a convincing replication? J Exp Soc Psychol 50:217-224.
77. Hajcak G, Foti D (2008) Errors are aversive: Defensive motivation and the error-related negativity. Psychol Sci 19(2):103-108.
78. Dai X, Wertenbroch K, Brendl CM (2008) The value heuristic in judgments of relative frequency. Psychol Sci 19(1):18-19.
79. Alter AL, Oppenheimer DM (2008) Effects of fluency on psychological distance and mental construal (or why New York is a large city, but New York is a civilized jungle). Psychol Sci 19(2):161-167.
80. Greene CS, Penrod NM, Williams SM, Moore JH (2009) Failure to replicate a genetic association may provide important clues about genetic architecture. PLoS One 4(6): e5639.
81. Djulbegovic B, Hozo I (2014) Effect of initial conditions on reproducibility of scientific research. Acta Inform Med 22(3):156-159.
82. Cesario J (2014) Priming, replication, and the hardest science. Perspect Psychol Sci 9(1): 40-48.
83. Henrich J, Heine SJ, Norenzayan A (2010) The weirdest people in the world? Behav Brain Sci 33(2-3):61-83, discussion 83-135.
84. Leek JT, Peng RD (2015) Statistics. What is the question? Science 347(6228):1314-1315.
85. Pashler H, Harris CR (2012) Is the replicability crisis overblown? Three arguments examined. Perspect Psychol Sci 7(6):531-536.
86. Dreber A, et al. (2015) Using prediction markets to estimate the reproducibility of scientific research. Proc Natl Acad Sci USA 112(50):15343-15347.
87. Monin B, et al. (2014) Commentaries and rejoinder on Klein et al. (2014). Soc Psychol 45(4):299-311.
88. Cacioppo JT, Berntson GG (1992) Social psychological contributions to the decade of the brain. Doctrine of multilevel analysis. Am Psychol 47(8):1019-1028.
89. Bevan W (1991) Contemporary psychology: A tour inside the onion. Am Psychol 46(5): 475-483.
90. Sattler JM (1970) Racial "experimenter effects" in experimentation, testing, interviewing, and psychotherapy. Psychol Bull 73(2):137-160.
91. Anderson CA (1989) Temperature and aggression: Ubiquitous effects of heat on occurrence of human violence. Psychol Bull 106(1):74-96.
92. May CP, Hasher L, Stoltzfus ER (1993) Optimal time of day and the magnitude of age differences in memory. Psychol Sci 4(5):326-330.