The reproducibility project: A model of large-scale collaboration for empirical research on reproducibility

Implementing Reproducible Research

Volumn , Issue , 2014, Pages 299-324

  Aarts, Alexander A ^a   Alexander, Anita ^b   Attridge, Peter ^c   Bahník, Štepán ^d   Barnett Cowan, Michael ^e   Bartmess, Elizabeth ^f   Bosco, Frank A ^g   Brown, Benjamin ^c   Brown, Kristina ^c   Chandler, Jesse J ^h   Clay, Russ ⁱ   Cleary, Hayley ^j   Cohn, Michael ^f   Costantini, Giulio ^k   Crusius, Jan ^l   DeCoster, Jamie ^b   DeGaetano, Michelle ^c   Donohue, Ryan ^m   Dunn, Elizabeth ⁿ   Eggleston, Casey ^b   Estel, Vivien ^o   Farach, Frank J ^p   Fiedler, Susann ^q   Field, James G ^g   Fitneva, Stanka ^r   Foster, Joshua D ^s   Frazier, Rebecca S ^b   Galliani, Elisa Maria ^t   Giner Sorolla, Roger ^u   Goss, R Justin ^v   Graham, Jesse ^w   Grange, James A ^x   Hartshorne, Joshua ^y   Hayes, Timothy B ^w   Hicks, Grace ^c   Humphries, Denise ^c   Jahn, Georg ^z   Johnson, Kate ^w   Joy Gaba, Jennifer A ^j   Goellner, Lars ^o   Kappes, Heather Barry ^aa   Lai, Calvin K ^b   Lakens, Daniel ^ab   Lane, Kristin A ^ac   LeBel, Etienne P ^e   Lee, Minha ^b   Lemm, Kristi ^ad   Lewis, Melissa ^ae   Lin, Stephanie C ^af   Mackinnon, Sean ^ag   Mainard, Heather ^c   Mann, Nathaniel ^ah   May, Michael ^ai   Motyl, Matt ^b   Moore, Katherine ^m   Müller, Stephanie M ^o   Nosek, Brian A ^b   Olsson, Catherine ^y   Perugini, Marco ^k   Pitts, Michael ^ae   Ratliff, Kate ^aj   Renkewitz, Frank ^o   Rutchick, Abraham M ^ah   Sandstrom, Gillian ⁿ   Selterman, Dylan ^ak   Simpson, William ^b   Smith, Colin Tucker ^aj   Spies, Jeffrey R ^b   Talhelm, Thomas ^b   van't Veer, Anna ^al   Vianello, Michelangelo ^t   more..

^a Syngaschem BV   (Netherlands)

^b UNIVERSITY OF VIRGINIA   (United States)

^c GEORGIA GWINNETT COLLEGE   (United States)

^d INSTITUTE OF PHYSIOLOGY   (Czech Republic)

^e UNIVERSITY OF WESTERN ONTARIO   (Canada)

^f UNIVERSITY OF CALIFORNIA   (United States)

^g MARSHALL UNIVERSITY   (United States)

^h PRIME Research   (United Kingdom)

ⁱ UNIVERSITY OF RICHMOND   (United States)

^j VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^k UNIVERSITY OF MILANO BICOCCA   (Italy)

^l UNIVERSITY OF COLOGNE   (Germany)

^m ELMHURST COLLEGE   (United States)

ⁿ UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^o UNIVERSITY OF ERFURT   (Germany)

^p UNIVERSITY OF WASHINGTON   (United States)

^q MAX PLANCK INSTITUTE FOR RESEARCH ON COLLECTIVE GOODS   (Germany)

^r Queen's University   (Ireland)

^s UNIVERSITY OF SOUTH ALABAMA   (United States)

^t UNIVERSITY OF PADOVA   (Italy)

^u UNIVERSITY OF KENT   (United Kingdom)

^v UNIVERSITY OF TEXAS AT SAN ANTONIO   (United States)

^w UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^x KEELE UNIVERSITY   (United Kingdom)

^y MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^z UNIVERSITY OF GREIFSWALD   (Germany)

^aa LONDON SCHOOL OF ECONOMICS   (United Kingdom)

^ab EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^ac BARD COLLEGE   (United States)

^ad WESTERN WASHINGTON UNIVERSITY   (United States)

^ae REED COLLEGE   (United States)

^af STANFORD UNIVERSITY   (United States)

^ag DALHOUSIE UNIVERSITY   (Canada)

^ah CALIFORNIA STATE UNIVERSITY   (United States)

^ai UNIVERSITY OF BONN   (Germany)

^aj UNIVERSITY OF FLORIDA   (United States)

^ak UNIVERSITY OF MARYLAND   (United States)

^al TILBURG UNIVERSITY   (Netherlands)

more..

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84982717820     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1201/b16868     Document Type: Chapter

References (53)

1. Amir, Y. and Sharon, I. (1990). Replication research: A “must” for the scientific advancement of psychology. Journal of Social Behavior and Personality, 5, 51–69.
2. Baillargeon, R., Spelke, E. S., and Wasserman, S. (1985). Object permanence in five-month-old infants. Cognition, 20, 191–208.
3. Bauer, H. H. (1992). Scientific Literacy and the Myth of the Scientific Method. Chicago, IL: University of Illinois Press.
4. Begley, C. G. and Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483, 531–533. doi:10.1038/483531a.
5. Belding, T. C. (2000). Numerical replication of computer simulations: Some pitfalls and how to avoid them. arXiv preprint nlin/0001057.
6. Campbell, D. T., Stanley, J. C., and Gage, N. L. (1963). Experimental and Quasi-Experimental Designs for Research (pp. 171–246). Boston, MA: Houghton Mifflin.
7. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
8. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
9. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159.
10. Collins, H. M. (1985). Changing Order. London, U.K. Sage.
11. Estellés-Arolas, E. and González-Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38(2), 189–200.
12. Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5(4), e10068. doi:10.1371/journal.pone.0010068.
13. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. doi:10.1007/s1192-011-0494-7.
14. Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin and Review, 19, 975–991.
15. Fuchs, H., Jenny, M., and Fiedler, S. (2012). Psychologists are open to change, yet wary of rules. Perspectives on Psychological Science, 7, 634–637. doi:10.1177/1745691612459521.
16. Giner-Sorolla, R. (2012). Science or art? How esthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571. doi:10.1177/1745691612457576.
17. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
18. Hartshorne, J. K. and Schachner, A. (2012a). Tracking replicability as a method of post-publication open evaluation. Frontiers in Computational Neuroscience, 6(8), 1–13. doi:10.3389/fncom.2012.0008.
19. Hartshorne, J. K. and Schachner, A. (2012b). Where’s the data? The Psychologist, 25, 355.
20. Hempel, C. G. and Oppenheim, P. (1948). Studies in the logic of explanation. Philosophy of Science, 15, 135–175.
21. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124.
22. Ioannidis, J. P. A., Ntzani, E. E., Trikalinos, T. A., and ContopoulosIoannidis, D. G. (2001). Replication validity of genetic association studies. Nature Genetics, 29, 306–309.
23. Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59, 434–446. doi: 10.1016/j.bbr.2011.03.031.
24. John, L., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. doi: 10.1177/0956797611430953.
25. Koole, S. L. and Lakens, D. (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7, 608–614. doi:10.1177/1745691612462586.
26. Kuhn, T. S. (1962). The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.
27. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498. doi:10.1037/0033-2909.108.3.480.
28. LeBel, E. P. and Paunonen, S. V. (2011). Sexy but often unreliable: Impact of unreliability on the replicability of experimental findings involving implicit measures. Personality and Social Psychology Bulletin, 37, 570–583.
29. LeBel, E. P. and Peters, K. R. (2011). Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15, 371–379. doi:10.1037/a0025172.
30. Ledgerwood, A. and Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7, 60–66. doi:10.1177/1745691611427304.
31. Lipsey, M. W. and Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209. doi:10.1037/0003-066X.48.12.1181.
32. Mahoney, M. J. (1985). Open exchange and epistemic process. American Psychologist, 40, 29–39.
33. Makel, M. C., Plucker, J. A., and Hagerty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542. doi:10.1177/1745691612460688.
34. Morrell, K. and Lucas, J. W. (2012). The replication problem and its implications for policy studies. Critical Policy Studies, 6, 182–200. doi:10.1080/19460171.2012.689738.
35. Neuliep, J. W. and Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior and Personality, 5, 85–90.
36. Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220.
37. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5(2), 241–301. doi:10.1037/1082-989X.5.2.241.
38. Nielson, M. (2011). Reinventing discovery: The new era of networked science. Princeton University Press.
39. Nosek, B. A. and Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217–243. doi:1080/1047840X.2012.692215.
40. Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058.
41. Open Science Collaboration. (2012). An open, large-scale collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660. doi:10.1177/1745691612462588.
42. Piaget, J. (1952). The Origins of Intelligence in Children. New York: International University Press.
43. Piaget, J. (1954). The Construction of Reality in the Child. New York: Basic.
44. Popper, K. (1934/1992). The Logic of Scientific Discovery. New York: Routledge.
45. Prinz, F., Schlange, T., and Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712–713. doi:10.1038/nrd3439-c1.
46. Rosenthal, R. and Rosnow, R. L. (1960). Artifact in Behavioral Research. New York: Academic Press.
47. Salmon, W. (1989). Four Decades of Scientific Explanation. Minneapolis, MN: University of Minnesota Press.
48. Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological Methods, 17, 551–566.
49. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. doi:10.1037/a0015108.
50. Sedlmeier, P. and Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316.
51. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632.
52. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—Or vice versa. Journal of the American Statistical Association, 54, 30–34.
53. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives in Psychological Science, 4, 274–90. doi:10.1111/j.1745-6924.2009.01125.x.