Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Nature Human Behaviour

Volumn 2, Issue 9, 2018, Pages 637-644

  Camerer, Colin F ^a   Dreber, Anna ^b   Holzmeister, Felix ^c   Ho, Teck Hua ^d   Huber, Jürgen ^c   Johannesson, Magnus ^b   Kirchler, Michael ^c,e   Nave, Gideon ^f   Nosek, Brian A ^g,h   Pfeiffer, Thomas ⁱ   Altmejd, Adam ^b   Buttrick, Nick ^g,h   Chan, Taizan ^j   Chen, Yiling ^k   Forsell, Eskil ^l   Gampa, Anup ^g,h   Heikensten, Emma ^b   Hummer, Lily ^h   Imai, Taisuke ^m   Isaksson, Siri ^b   Manfredi, Dylan ^f   Rose, Julia ^c   Wagenmakers, Eric Jan ⁿ   Wu, Hang ^o   more..

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^b STOCKHOLM SCHOOL OF ECONOMICS   (Sweden)

^c UNIVERSITY OF INNSBRUCK   (Austria)

^d NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^e UNIVERSITY OF GOTHENBURG   (Sweden)

^f UNIVERSITY OF PENNSYLVANIA   (United States)

^g UNIVERSITY OF VIRGINIA   (United States)

^h CENTER FOR OPEN SCIENCE   (United States)

ⁱ New Zealand Institute for Advanced Studies   (New Zealand)

^j NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^k HARVARD UNIVERSITY   (United States)

^l SPOTIFY   (United Kingdom)

^m UNIVERSITY OF MUNICH   (Germany)

ⁿ UNIVERSITY OF AMSTERDAM   (Netherlands)

^o HARBIN INSTITUTE OF TECHNOLOGY   (China)

more..

Author keywords

[No Author keywords available]

Indexed keywords

BAYES THEOREM; HUMAN; PROCEDURES; PUBLICATION; REPRODUCIBILITY; RESEARCH; SAMPLE SIZE; SOCIOLOGY; STATISTICS AND NUMERICAL DATA;

BAYES THEOREM; HUMANS; PERIODICALS AS TOPIC; REPRODUCIBILITY OF RESULTS; RESEARCH; SAMPLE SIZE; SOCIAL SCIENCES;

EID: 85052962253     PISSN: None     EISSN: 23973374     Source Type: Journal    
DOI: 10.1038/s41562-018-0399-z     Document Type: Article

References (56)

1. McNutt, M. Reproducibility. Science 343, 229 (2014)
2. Baker, M. Is there a reproducibility crisis? Nature 533, 452–454 (2016)
3. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017)
4. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005)
5. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 (2011)
6. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012)
7. Lawrence, M. S. et al. Mutational heterogeneity in cancer and the search for new cancer-associated genes. Nature 499, 214–218 (2013)
8. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)
9. Maniadis, Z., Tufano, F. & List, J. A. One swallow doesn’t make a summer: new evidence on anchoring effects. Am. Econ. Rev. 104, 277–290 (2014)
10. Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015)
11. Klein, R. A. et al. Investigating variation in replicability: a ‘many labs’ replication project. Soc. Psychol. 45, 142–152 (2014)
12. Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015)
13. Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351, 1433–1436 (2016)
14. Ebersole, C. R. et al. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J. Exp. Soc. Psychol. 67, 68–82 (2016)
15. Klein, R. A. et al. Many Labs 2: investigating variation in replicability across sample and setting. Adv. Methods Prac. Psychol. Sci. (in the press)
16. Ackerman, J. M., Nocera, C. C. & Bargh, J. A. Incidental haptic sensations influence social judgments and decisions. Science 328, 1712–1715 (2010)
17. Aviezer, H., Trope, Y. & Todorov, A. Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338, 1225–1229 (2012)
18. Balafoutas, L. & Sutter, M. Affirmative action policies promote women and do not harm efficiency in the laboratory. Science 335, 579–582 (2012)
19. Derex, M., Beugin, M.-P., Godelle, B. & Raymond, M. Experimental evidence for the influence of group size on cultural complexity. Nature 503, 389–391 (2013)
20. Duncan, K., Sadanand, A. & Davachi, L. Memory’s penumbra: episodic memory decisions induce lingering mnemonic biases. Science 337, 485–487 (2012)
21. Gervais, W. M. & Norenzayan, A. Analytic thinking promotes religious disbelief. Science 336, 493–496 (2012)
22. Gneezy, U., Keenan, E. A. & Gneezy, A. Avoiding overhead aversion in charity. Science 346, 632–635 (2014)
23. Hauser, O. P., Rand, D. G., Peysakhovich, A. & Nowak, M. A. Cooperating with the future. Nature 511, 220–223 (2014)
24. Janssen, M. A., Holahan, R., Lee, A. & Ostrom, E. Lab experiments for the study of social-ecological systems. Science 328, 613–617 (2010)
25. Karpicke, J. D. & Blunt, J. R. Retrieval practice produces more learning than elaborative studying with concept mapping. Science 331, 772–775 (2011)
26. Kidd, D. C. & Castano, E. Reading literary fiction improves theory of mind. Science 342, 377–380 (2013)
27. Kovacs, Á. M. & Téglás, E. & Endress, A. D. The social sense: susceptibility to others’ beliefs in human infants and adults. Science 330, 1830–1834 (2010)
28. Lee, S. W. S. & Schwarz, N. Washing away postdecisional dissonance. Science 328, 709 (2010)
29. Morewedge, C. K., Huh, Y. E. & Vosgerau, J. Thought for food: imagined consumption reduces actual consumption. Science 330, 1530–1533 (2010)
30. Nishi, A., Shirado, H., Rand, D. G. & Christakis, N. A. Inequality and visibility of wealth in experimental social networks. Nature 526, 426–429 (2015)
31. Pyc, M. A. & Rawson, K. A. Why testing improves memory: mediator effectiveness hypothesis. Science 330, 335 (2010)
32. Ramirez, G. & Beilock, S. L. Writing about testing worries boosts exam performance in the classroom. Science 331, 211–213 (2011)
33. Rand, D. G., Greene, J. D. & Nowak, M. A. Spontaneous giving and calculated greed. Nature 489, 427–430 (2012)
34. Shah, A. K., Mullainathan, S. & Shafir, E. Some consequences of having too little. Science 338, 682–685 (2012)
35. Sparrow, B., Liu, J. & Wegner, D. M. Google effects on memory: cognitive consequences of having information at our fingertips. Science 333, 776–778 (2011)
36. Wilson, T. D. et al. Just think: the challenges of the disengaged mind. Science 345, 75–77 (2014)
37. Bohannon, J. Replication effort provokes praise—and ‘bullying’ charges. Science 344, 788–789 (2014)
38. Gilbert, D. T., King, G., Pettigrew, S. & Wilson, T. D. Comment on "Estimating the reproducibility of psychological science". Science 351, 1037 (2016)
39. Anderson, C. J. et al. Response to comment on "Estimating the reproducibility of psychological science". Science 351, 1037 (2016)
40. Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008)
41. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011)
42. Etz, A. & Vandekerckhove, J. A Bayesian perspective on the Reproducibility Project: Psychology. PLoS One 11, e0149794 (2016)
43. Gelman, A. & Stern, H. The difference between “significant” and “not significant” is not itself statistically significant. Am. Stat. 60, 328–331 (2006)
44. Cumming, G. Replication and P intervals: P values predict the future only vaguely, but confidence intervals do much better. Psychol. Sci. 3, 286–300 (2008)
45. Verhagen, J. & Wagenmakers, E.-J. Bayesian tests to quantify the result of a replication attempt. J. Exp. Psychol. Gen. 143, 1457–1475 (2014)
46. Simonsohn, U. Small telescopes: detectability and the evaluation of replication results. Psychol. Sci. 26, 559–569 (2015)
47. Patil, P., Peng, R. D. & Leek, J. T. What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11, 539–544 (2016)
48. Wagenmakers, E.-J. et al. Bayesian inference for psychology. Part II: example applications with JASP. Psychon. Bull. Rev. 25, 58–76 (2017)
49. Lee, M. D. & Wagenmakers, E.-J. Bayesian Cognitive Modeling: A Practical Course (Cambridge Univ. Press, Cambridge, 2013)
50. Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112, 15343–15347 (2015)
51. Benjamin, D. et al. Redefine statistical significance. Nat. Hum. Behav. 2, 6–10 (2018)
52. Jeffreys, H. Theory of Probability (Oxford Univ. Press, Oxford, 1961)
53. Kass, R. E. & Raftery, A. E. Bayes factors. J. Am. Stat. Assoc. 90, 773–795 (1995)
54. Arrow, K. J. et al. The promise of prediction markets. Science 320, 877–878 (2008)
55. Nosek, B. A., Ebersole, C. R., DeHaven, A. & Mellor, D. M. The preregistration revolution. Proc. Natl Acad. Sci. USA 115, 2600–2606 (2018)
56. Nosek, B. A. et al. Promoting an open research culture: author guidelines for journals could help to promote transparency, openness, and reproducibility. Science 348, 1422–1425 (2015)