A Bayesian perspective on the reproducibility project: Psychology

PLoS ONE

Volumn 11, Issue 2, 2016, Pages

  Etz, Alexander ^a   Vandekerckhove, Joachim ^b,c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c Ringgold Standard Institution Informatics   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADOPTION; ALTERNATIVE HYPOTHESIS; EFFECT SIZE; HUMAN; MAJOR CLINICAL STUDY; NULL HYPOTHESIS; PSYCHOLOGY; PUBLISHING; REPRODUCIBILITY; SAMPLE SIZE; BAYES THEOREM; METHODOLOGY;

BAYES THEOREM; HUMANS; PSYCHOLOGY; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE;

EID: 84960429337     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0149794     Document Type: Article

References (38)

1. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015; 349(6251):aac4716. doi: 10.1126/science.aac4716 PMID: 26315443
2. Handwerk B. Scientists replicated 100 psychology studies, and fewer than half got the same results; 2015. Accessed: 2015-10-31. http://bit.ly/1OYZVHY
3. Jump P. More than half of psychology papers are not reproducible; 2015. Accessed: 2015-10-31. http://bit.ly/1GwLHGh
4. Connor S. Study reveals that a lot of psychology research really is just 'psycho-babble'; 2015. Accessed: 2015-10-31. http://ind.pn/1R07hby
5. Feldman Barrett L. Psychology is not in crisis; 2015. Accessed: 2015-10-31. http://nyti.ms/1PInTEg
6. American Psychological Association. Science paper shows low replicability of psychology studies; 2015. Accessed: 2015-10-31. http://bit.ly/1NogtsD
7. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013; 14 (5):365-376. doi: 10.1038/nrn3475 PMID: 23571845
8. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance- or vice versa. Journal of the American Statistical Association. 1959; 54:30-34. doi: 10.1080/01621459.1959.10501497
9. Rosenthal R. The file drawer problem and tolerance for null results. Psychological bulletin. 1979; 86 (3):638. doi: 10.1037/0033-2909.86.3.638
10. Pashler H, Wagenmakers EJ. Editors' introduction to the Special Section on Replicability in Psychological Science: A crisis of confidence? Perspectives on Psychological Science. 2012; 7:528-530. doi: 10.1177/1745691612465253 PMID: 26168108
11. Ioannidis JP. Why most discovered true associations are inflated. Epidemiology. 2008; 19(5):640-648. doi: 10.1097/EDE.0b013e31818131e7 PMID: 18633328
12. Bakan D. The test of significance in psychological research. Psychological Bulletin. 1966; 66:423-437. doi: 10.1037/h0020412 PMID: 5974619
13. Walster GW, Cleary TA. A proposal for a new editorial policy in the social sciences. The American Statistician. 1970; 24(2):16-19. doi: 10.2307/2681924
14. Guan M, Vandekerckhove J. A Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review. 2016; 23(1):74-86. http://escholarship.org/uc/item/2682p4tr doi: 10.3758/s13423-015- 0868-6
15. Etz A, Wagenmakers EJ. Origin of the Bayes Factor. Preprint available: arXiv:151108180 [statOT]. 2015;Available from: http://arxiv.org/abs/1511.08180
16. Kass RE, Raftery AE. Bayes Factors. Journal of the American Statistical Association. 1995; 90:773-795. doi: 10.1080/01621459.1995.10476572
17. Rouder JN, Speckman PL, Sun D, Morey RD, Iverson G. Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review. 2009; 16:225-237. doi: 10.3758/PBR.16.2.225
18. Verhagen AJ, Wagenmakers EJ. Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General. 2014; 143:1457-1475. doi: 10.1037/a0036731
19. Etz A. The Bayesian Reproducibility Project; 2015. Accessed: 2015-10-31. http://bit.ly/1U1YO9J
20. Jeffreys H. Theory of Probability. 1st ed. Oxford, UK: Oxford University Press; 1939.
21. Young NS, Ioannidis JP, Al-Ubaydli O. Why current publication practices may distort science. PLoS Medicine. 2008; 5(10):e201. doi: 10.1371/journal.pmed.0050201 PMID: 18844432
22. Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods. 2014; 5:60-78. doi: 10.1002/jrsm.1095 PMID: 26054026
23. Abramowitz M, Stegun IA, et al. Handbook of mathematical functions. vol. 1. Dover New York; 1972.
24. Wetzels R, Matzke D, Lee MD, Rouder JN, Iverson GJ, Wagenmakers EJ. Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science. 2011; 6:291-298. doi: 10.1177/1745691611406923 PMID: 26168519
25. Johnson VE. Revised standards for statistical evidence. Proceedings of the National Academy of Sciences. 2013; 110(48):19313-19317. doi: 10.1073/pnas.1313476110
26. Maxwell SE, Lau MY, Howard GS. Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? American Psychologist. 2015; 70(6):487. doi: 10.1037/a0039400 PMID: 26348332
27. Ioannidis JPA. Why most published research findings are false. PLoS Medicine. 2005; 2:696-701. doi: 10.1371/journal.pmed.0020124
28. Fraley RC, Vazire S. The N-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE. 2014; 9:e109019. doi: 10.1371/journal.pone.0109019 PMID: 25296159
29. Berger JO, Delampady M. Testing precise hypotheses. Statistical Science. 1987; 2:317-352. doi: 10.1214/ss/1177013245
30. Berger JO, Sellke T. Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association. 1987; 82:112-139. doi: 10.1080/01621459.1987. 10478397
31. Edwards W, Lindman H, Savage LJ. Bayesian Statistical Inference for Psychological Research. Psychological Review. 1963; 70:193-242. doi: 10.1037/h0044139
32. Goodman S. A dirty dozen: Twelve p-value misconceptions. In: Seminars in hematology. vol. 45. Elsevier; 2008. p. 135-140.
33. Rouder JN. Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review. 2014; 21 (2):301-308. doi: 10.3758/s13423-014-0595-4
34. Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HLJ, Kievit RA. An agenda for purely confirmatory research. Perspectives on Psychological Science. 2012; 7:627-633. doi: 10.1177/1745691612463078
35. Matzke D, Nieuwenhuis S, van Rijn H, Slagter HA, van der Molen MW, Wagenmakers EJ. The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General. 2015; 144:e1-e15. doi: 10.1037/xge0000038
36. Schönbrodt FD, Wagenmakers EJ, Zehetleitner M, Perugini M. Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods. in press;.
37. Love J, Selker R, Marsman M, Jamil T, Dropmann D, Verhagen AJ, et al. JASP [Computer software]; 2015. https://jasp-stats.org/
38. Marsman M, Dablander F, Baribault B, Etz A, Gronau QF, Jamil T, et al. A Bayesian reanalysis of the reproducibility project.; 2015. Manuscript in preparation.