The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 55-67

  Schweinsberg, Martin ^a   Madan, Nikhil ^a   Vianello, Michelangelo ^b   Sommer, S Amy ^c   Jordan, Jennifer ^d   Tierney, Warren ^a   Awtrey, Eli ^e   Zhu, Luke Lei ^f   Diermeier, Daniel ^g   Heinze, Justin E ^h   Srinivasan, Malavika ⁱ   Tannenbaum, David ^g   Bivolaru, Eliza ^a   Dana, Jason ^j   Davis Stober, Clintin P ^k   du Plessis, Christilene ^l   Gronau, Quentin F ^m   Hafenbrack, Andrew C ⁿ   Liao, Eko Yi ^o   Ly, Alexander ^m   Marsman, Maarten ^m   Murase, Toshio ^p   Qureshi, Israr ^q   Schaerer, Michael ^a   Thornley, Nico ^a   Tworek, Christina M ^r   Wagenmakers, Eric Jan ^m   Wong, Lynn ^a   Anderson, Tabitha ^s   Bauman, Christopher W ^t   Bedwell, Wendy L ^u   Brescoll, Victoria ^j   Canavan, Andrew ^s   Chandler, Jesse J ^v   Cheries, Erik ^w   Cheryan, Sapna ^e   Cheung, Felix ^x,y   Cimpian, Andrei ^r   Clark, Mark A ^z   Cordon, Diana ^s   Cushman, Fiery ⁱ   Ditto, Peter H ^t   Donahue, Thomas ^s   Frick, Sarah E ^u   Gamez Djokic, Monica ^aa   Grady, Rebecca Hofstein ^t   Graham, Jesse ^ab   Gu, Jun ^ac   Hahn, Adam ^ad   Hanson, Brittany E ^ae   Hartwich, Nicole J ^ad   Hein, Kristie ^s   Inbar, Yoel ^af   Jiang, Lily ^e   Kellogg, Tehlyr ^s   Kennedy, Deanna M ^ag   Legate, Nicole ^s   Luoma, Timo P ^ad   Maibuecher, Heidi ^s   Meindl, Peter ^ab   Miles, Jennifer ^t   Mislin, Alexandra ^z   Molden, Daniel C ^aa   Motyl, Matt ^ae   Newman, George ^j   Ngo, Hoai Huong ^ah   Packham, Harvey ^y   Ramsay, Philip S ^u   Ray, Jennifer L ^ai   Sackett, Aaron M ^aj   Sellier, Anne Laure ^c   Sokolova, Tatiana ^c,h   Sowden, Walter ^h   Storage, Daniel ^r   Sun, Xiaomin ^ak   Van Bavel, Jay J ^ai   Washburn, Anthony N ^ae   Wei, Cong ^ak   Wetter, Erik ^al   Wilson, Carlos T ^s   Darroux, Sophie Charlotte ^a   Uhlmann, Eric Luis ^a   more..

^a INSEAD   (Singapore)

^b UNIVERSITY OF PADOVA   (Italy)

^c HEC PARIS   (France)

^d UNIVERSITY OF GRONINGEN   (Netherlands)

^e UNIVERSITY OF WASHINGTON   (United States)

^f UNIVERSITY OF MANITOBA   (Canada)

^g UNIVERSITY OF CHICAGO   (United States)

^h UNIVERSITY OF MICHIGAN   (United States)

ⁱ HARVARD UNIVERSITY   (United States)

^j YALE UNIVERSITY   (United States)

^k UNIVERSITY OF MISSOURI   (United States)

^l ERASMUS UNIVERSITY   (Netherlands)

^m UNIVERSITY OF AMSTERDAM   (Netherlands)

ⁿ CATÓLICA LISBON SCHOOL OF BUSINESS AND ECONOMICS   (Portugal)

^o MACAU UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Macao)

^p ROOSEVELT UNIVERSITY   (United States)

^q IE BUSINESS SCHOOL   (Spain)

^r UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^s ILLINOIS INSTITUTE OF TECHNOLOGY   (United States)

^t UNIVERSITY OF CALIFORNIA   (United States)

^u UNIVERSITY OF SOUTH FLORIDA   (United States)

^v MATHEMATICA POLICY RESEARCH   (United States)

^w UNIVERSITY OF MASSACHUSETTS   (United States)

^x MICHIGAN STATE UNIVERSITY   (United States)

^y UNIVERSITY OF HONG KONG   (Hong Kong)

^z AMERICAN UNIVERSITY   (United States)

^aa NORTHWESTERN UNIVERSITY   (United States)

^ab UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^ac MONASH UNIVERSITY   (Australia)

^ad UNIVERSITY OF COLOGNE   (Germany)

^ae UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

^af UNIVERSITY OF TORONTO   (Canada)

^ag UNIVERSITY OF WASHINGTON BOTHELL   (United States)

^ah UNIVERSITÉ PARIS OUEST NANTERRE LA DÉFENSE   (France)

^ai NEW YORK UNIVERSITY   (United States)

^aj UNIVERSITY OF ST THOMAS   (United States)

^ak BEIJING NORMAL UNIVERSITY   (China)

^al STOCKHOLM SCHOOL OF ECONOMICS   (Sweden)

more..

Author keywords

Crowdsourcing science; Meta science; Methodology; Replication; Reproducibility; Research transparency

Indexed keywords

CROWDSOURCING; EFFECT SIZE; MORALITY; PIPELINE; PUBLICATION; REPRODUCIBILITY;

EID: 84977144634     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.10.001     Document Type: Article

References (96)

1. Alogna, V.K., Attaya, M.K., Aucoin, P., Bahník, Š., Birch, S., Birt, A.R., et al. Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science 9:5 (2014), 556–578.
2. Azar, O.H., The social norm of tipping: A review. Journal of Applied Social Psychology 37:2 (2007), 380–402.
3. Begley, C.G., Ellis, L.M., Drug development: Raise standards for preclinical cancer research. Nature 483 (2012), 531–533.
4. Berinsky, A., Margolis, M.F., Sances, M.W., Can we turn shirkers into workers?. Journal of Experimental Social Psychology 66 (2016), 20–28.
5. Blendon, R.J., Benson, J.M., Brodie, M., Morin, R., Altman, D.E., Gitterman, G., Brossard, M., James, M., Bridging the gap between the public's and economists' views of the economy. Journal of Economic Perspectives 11 (1997), 105–118.
6. Bohannon, J., Replication effort provokes praise—And ‘bullying’ charges. Science 344 (2014), 788–789.
7. Borenstein, M., Hedges, L.V., Higgins, J.P., Rothstein, H.R., Introduction to meta-analysis. 2011, John Wiley & Sons.
8. Brandt, M.J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., Grange, J.A., Perugini, M., Spies, J., van 't Veer, A., The replication recipe: What makes for a convincing replication?. Journal of Experimental Social Psychology 50 (2014), 217–224.
9. Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., Munafo, M.R., Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14 (2013), 1–12.
10. Caplan, B., What makes people think like economists? Evidence on economic cognition from the ‘Survey of Americans and Economists on the Economy’. Journal of Law and Economics 43 (2001), 395–426.
11. Caplan, B., Systematically biased beliefs about economics: Robust evidence of judgmental anomalies from the ‘Survey of Americans and Economists on the Economy'. The Econometrics Journal 112 (2002), 433–458.
12. Clark, C.J., Chen, E.E., Ditto, P.H., Moral coherence processes: Constructing culpability and consequences. Current Opinion in Psychiatry, 2015 (in press).
13. Cochran, W.G., Carroll, S.P., A sampling investigation of the efficiency of weighting inversely as the estimated variance. Biometrics 9:4 (1953), 447–459.
14. Cohen, J., Statistical power analysis for the behavioral sciences. 2nd ed., 1988, Lawrence Erlbaum Associates, New Jersey.
15. Cooper, H., Hedges, L., (eds.) The handbook of research synthesis, 1994, Russell Sage Foundation, New York.
16. Cumming, G., Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science 3:4 (2008), 286–300.
17. Curran, P.G., Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology 66 (2016), 4–19.
18. Davis-Stober, C.P., Dana, J., Comparing the accuracy of experimental estimates to guessing: A new perspective on replication and the “crisis of confidence” in psychology. Behavior Research Methods 46 (2014), 1–14.
19. Diermeier, D., Reputation rules: Strategies for building your company's most valuable asset. 2011, McGraw-Hill.
20. Ditto, P.H., Lopez, D.F., Motivated skepticism: The use of differential decision criteria for preferred and nonpreferred conclusions. Journal of Personality and Social Psychology 63 (1992), 568–584.
21. Ebersole, C.R., Klein, R.A., Atherton, O.E., The Many Lab., 2014 https://osf.io/89vqh/.
22. Ebersole, C.R., Atherton, O.E., Belanger, A.L., Skulborstad, H.M. Nosek, B.A., Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 2015 (in press).
23. Fabrigar, L.R., Wegener, D.T., Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66 (2016), 68–80.
24. Fehr, R., Gelfand, M.J., When apologies work: How matching apology components to victims' self-construals facilitates forgiveness. Organizational Behavior and Human Decision Processes 113:1 (2010), 37–50.
25. Festinger, L., A theory of cognitive dissonance. 1957, Stanford University Press, Stanford, CA.
26. Gilbert, D., Some thoughts on shameless bullies. 2014 (Available at: http://www.wjh.harvard.edu/~dtg/Bullies.pdf).
27. Greenwald, A.G., Nosek, B.A., Banaji, M.R., Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology 85 (2003), 197–216.
28. Hales, A.H., Does the conclusion follow from the evidence? Recommendations for improving research. Journal of Experimental Social Psychology 66 (2016), 39–46.
29. Henrich, J., Heine, S.J., Norenzayan, A., The weirdest people in the world?. Behavioral and Brain Sciences 33 (2010), 61–83.
30. Hsee, C.K., Loewenstein, G.F., Blount, S., Bazerman, M.H., Preference reversals between joint and separate evaluation of options: A review and theoretical analysis. Psychological Bulletin 125:5 (1999), 576–590.
31. Hüffmeier, J., Mazei, J., Schultze, T., Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology 66 (2016), 81–92.
32. Ioannidis, J.P., Why most published research findings are false. PLoS Medicine, 2005 ( http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124).
33. Ioannidis, J.P.A., Trikalinos, T.A., An exploratory test for an excess of significant findings. Clinical Trials 4 (2007), 245–253.
34. Jeffreys, H., Theory of probability. 3rd ed., 1961, Oxford University Press, Oxford, UK.
35. Jordan, J., Diermeier, D.A., Galinsky, A.D., The strategic samaritan: How effectiveness and proximity affect corporate responses to external crises. Business Ethics Quarterly 22:04 (2012), 621–648.
36. Kahneman, D., A proposal to deal with questions about priming effects. Retrieved at: http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf, 2012.
37. Kahneman, D., A new etiquette for replication. Social Psychology 45:4 (2014), 310–311.
38. Kass, R.E., Raftery, A.E., Bayes factors. Journal of the American Statistical Association 90 (1995), 773–795.
39. Kitchin, R., The data revolution: Big data, open data, data infrastructures and their consequences. 2014, Sage.
40. Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B. Jr., Bahník, Š., Bernstein, M.J.…Nosek, B.A., Investigating variation in replicability: A “many labs” replication project. Social Psychology 45:3 (2014), 142–152.
41. Kunda, Z., The case for motivated reasoning. Psychological Bulletin 108 (1990), 480–498.
42. Lai, C.K., Marini, M., Lehr, S.A., Cerruti, C., Shin, J.L., Joy-Gaba, J.A.…Nosek, B.A., A comparative investigation of 17 interventions to reduce implicit racial preferences. Journal of Experimental Psychology: General 143 (2014), 1765–1785.
43. Landy, J., Uhlmann, E.L., Morality is personal. Invited submission to Graham, J., Gray, K., (eds.) The atlas of moral psychology, 2015.
44. Lewandowsky, S., Ecker, U.K.H., Seifert, C., Schwarz, N., Cook, J., Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest 13 (2012), 106–131.
45. Lieberman, M.D., Latitudes of acceptance. Available online at: http://edge.org/conversation/latitudes-of-acceptance, 2014.
46. Lipsey, M.W., Wilson, D.B., Practical meta-analysis (Vol. 49). 2001, Sage publications, Thousand Oaks, CA.
47. Liu, B., Ditto, P.H., What dilemma? Moral evaluation shapes factual belief. Social Psychological and Personality Science 4 (2013), 316–323.
48. Lord, C.G., Ross, L., Lepper, M.R., Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology 37 (1979), 2098–2109.
49. Ly, A., Verhagen, A.J., Wagenmakers, E.-J., Harold Jeffreys's default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology, 2015 (in press).
50. Mahoney, M.J., Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1:2 (1977), 161–175.
51. Maner, J.K., Into the wild: Field research can increase both replicability and real world impact. Journal of Experimental Social Psychology 66 (2016), 100–106.
52. Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H.A., van der Molen, M.W., Wagenmakers, E.-J., The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General 144 (2015), e1–e15.
53. Mitchell, J., On the emptiness of failed replications. 2014 (Available at: http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm).
54. Molden, D.C., Higgins, E.T., Motivated thinking. Holyoak, K., Morrison, B., (eds.) The Oxford Handbook of Thinking and Reasoning, 2012, Psychology Press, New York, 319–335.
55. Monin, B., Merritt, A., Moral hypocrisy, moral inconsistency, and the struggle for moral integrity. Mikulincer, M., Shaver, P.R., (eds.) The social psychology of morality: Exploring the causes of good and evil, 2012, American Psychological Association, Washington, DC.
56. Nelson, J.D., Finding useful questions: On Bayesian diagnosticity, probability, impact, and information gain. Psychological Review 112 (2005), 979–999.
57. Nelson, J.D., McKenzie, C.R.M., Cottrell, G.W., Sejnowski, T.J., Experience matters: Information acquisition optimizes probability gain. Psychological Science 7 (2010), 960–969.
58. Nickerson, R.S., How we know—and sometimes misjudge—what others know: Imputing one's own knowledge to others. Psychological Bulletin, 125(6), 1999, 737.
59. Nisbett, R.E., Peng, K., Choi, I., Norenzayan, A., Culture and systems of thought: Holistic vs. analytic cognition. Psychological Review 108 (2001), 291–310.
60. Norenzayan, A., Heine, S.J., Psychological universals: What are they and how can we know?. Psychological Bulletin 135 (2005), 763–784.
61. Nosek, B.A., Bar-Anan, Y., Scientific utopia: I. Opening scientific communication. Psychological Inquiry 23 (2012), 217–243.
62. Nosek, B.A., Lakens, D., Registered reports: A method to increase the credibility of published results. Social Psychology 45 (2014), 137–141.
63. Nosek, B.A., Spies, J.R., Motyl, M., Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7 (2012), 615–631.
64. Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251), 2015, 10.1126/science.aac4716.
65. Pace, K.M., Fediuk, T.A., Botero, I.C., The acceptance of responsibility and expressions of regret in organizational apologies after a transgression. Corporate Communications: An International Journal 15:4 (2010), 410–427.
66. Pizarro, D.A., Tannenbaum, D., Bringing character back: How the motivation to evaluate character influences judgments of moral blame. Shaver, P., Mikulincer, M., (eds.) The social psychology of morality: Exploring the causes of good and evil, 2011, APA books, New York.
67. Pizarro, D.A., Tannenbaum, D., Uhlmann, E.L., Mindless, harmless, and blameworthy. Psychological Inquiry 23 (2012), 185–188.
68. Prinz, F., Schlange, T., Asadullah, K., Believe it or not: How much can we rely on published data on potential drug targets?. Nature Reviews Drug Discovery 10 (2011), 712–713.
69. Rouder, J.N., Speckman, P.L., Sun, D., Morey, R.D., Iverson, G., Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review 16 (2009), 225–237.
70. Sakaluk, J.K., Exploring small, confirming big: An alternative system to the new statistics for advancing cumulative and replicable psychological research. Journal of Experimental Social Psychology 66 (2016), 47–54.
71. Schaller, M., The empirical benefits of conceptual rigor: Systematic articulation of conceptual hypotheses can reduce the risk of non-replicable results (and facilitate novel discoveries too). Journal of Experimental Social Psychology 66 (2016), 107–115.
72. Schnall, S., An experience with a registered replication project. Available at: http://www.psychol.cam.ac.uk/cece/blog#anchor-1, 2014.
73. Schnall, S., Further thoughts on replications, ceiling effects and bullying. Available at: http://www.psychol.cam.ac.uk/cece/blog, 2014.
74. Schnall, S., Social media and the crowd-sourcing of social psychology. Available at: http://www.psychol.cam.ac.uk/cece/blog, 2014.
75. Schooler, J., Unpublished results hide the decline effect. Nature, 470, 2011, 437.
76. Schooler, J., Metascience could rescue the ‘replication crisis’. Nature, 515, 2014, 9.
77. Schwarz, N., Strack, F., Does merely going through the same moves make for a “direct” replication? Concepts, contexts, and operationalizations. Social Psychology 45:3 (2014), 299–311.
78. Sears, D.O., College sophomores in the laboratory: Influences of a narrow database on social psychology's view of human nature. Journal of Personality and Social Psychology 51 (1986), 515–530.
79. Shweder, R.A., Haidt, J., The future of moral psychology: Truth, intuition, and the pluralist way. Psychological Science 4:6 (1993), 360–365.
80. Simmons, J., Nelson, L., Simonsohn, U., False-positive psychology: Undisclosed flexibility in data collection and analysis allow presenting anything as significant. Psychological Science 22:11 (2011), 1359–1366.
81. Simmons, J., Nelson, L., Simonsohn, U., Life after p-hacking. Presentation at the Annual Meeting of the Society for Personality and Social Psychology, 2013 (Available at: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2205186).
82. Simonsohn, U., Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26 (2015), 559–569.
83. Stanley, D.J., Spence, J.R., Expectations for replications: Are you realistic?. Perspectives on Psychological Science 9 (2014), 305–318.
84. Stroebe, W., Are most published social psychological findings false?. Journal of Experimental Social Psychology 66 (2016), 134–144.
85. Stroebe, W., Strack, F., The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9:1 (2014), 59–71.
86. Tannenbaum, D., Uhlmann, E.L., Diermeier, D., Moral signals, public outrage, and immaterial harms. Journal of Experimental Social Psychology 47 (2011), 1249–1254.
87. Tversky, A., Kahneman, D., Belief in the law of small numbers. Psychological Bulletin 76:2 (1971), 105–110.
88. Uhlmann, E.L., Zhu, L., Tannenbaum, D., When it takes a bad person to do the right thing. Cognition 126 (2013), 326–334.
89. Uhlmann, E.L., Pizarro, D., Diermeier, D., A person-centered approach to moral judgment. Perspectives on Psychological Science 10 (2015), 72–81.
90. Valdesolo, P., DeSteno, D., Moral hypocrisy: Social groups and the flexibility of virtue. Psychological Science 18:8 (2007), 689–690.
91. Verhagen, A.J., Wagenmakers, E.-J., Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General 143 (2014), 1457–1475.
92. Wagenmakers, E.-J., Grünwald, P., Steyvers, M., Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology 50 (2006), 149–166.
93. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L.J., Kievit, R.A., An agenda for purely confirmatory research. Perspectives on Psychological Science 7 (2012), 627–633.
94. Wagenmakers, E.-J., Verhagen, A.J., Ly, A., How to quantify the evidence for the absence of a correlation. Behavior Research Methods, 2015 (in press).
95. Wilson, T., Is there a crisis of false negatives in psychology?. Available at: http://timwilsonredirect.wordpress.com/2014/06/15/is-there-a-crisis-of-false-negatives-in-psychology/, 2014.
96. Yuill, N., Perner, J., Pearson, A., Peerbhoy, D., Ende, J., Children's changing understanding of wicked desires: From objective to subjective and moral. British Journal of Developmental Psychology 14:4 (1996), 457–475.