Bayesian evaluation of effect size after replicating an original study

PLoS ONE

Volumn 12, Issue 4, 2017, Pages

  Van Aert, Robbie C M ^a   Van Assen, Marcel A L M ^a,b  

^a TILBURG UNIVERSITY   (Netherlands)

^b UTRECHT UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ECONOMICS; EFFECT SIZE; HUMAN; META ANALYSIS; MODEL; NULL HYPOTHESIS; POWER ANALYSIS; PROBABILITY; PSYCHOLOGY; PUBLISHING; REPRODUCIBILITY; SAMPLE SIZE; BAYES THEOREM; META ANALYSIS (TOPIC);

BAYES THEOREM; META-ANALYSIS AS TOPIC; PUBLICATION BIAS; REPRODUCIBILITY OF RESULTS;

EID: 85017171096     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0175302     Document Type: Article

References (33)

1. Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012; 90(3): 891-904.
2. Fanelli D. Do pressures to publish increase scientists' bias? An empirical support from US States Data. PLoS ONE. 2010; 5(4).
3. Sterling TD, Rosenbaum WL, Weinkam JJ. Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician. 1995; 49(1): 108-12.
4. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012; 483(7391): 531-3. https://doi.org/10.1038/483531a PMID: 22460880
5. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015; 349(6251).
6. Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science. 2016; 351(6280): 1433-6. https://doi.org/10.1126/science.aaf0918 PMID: 26940865
7. Gilbert DT, King G, Pettigrew S, Wilson TD. Comment on "Estimating the reproducibility of psychological science". 2016; 351(6277).
8. Maxwell SE, Lau MY, Howard GS. Is psychology suffering from a replication crisis? What does "failure to replicate" really mean? American Psychologist. 2015; 70(6): 487-98. http://dx.doi.org/10.1037/a0039400. PMID: 26348332
9. Anderson SF, Maxwell SE. There's more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods. 2016; 21(1): 1-12. https://doi.org/10.1037/met0000051 PMID: 26214497
10. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to meta-analysis. Chichester, UK: John Wiley & Sons, Ltd.; 2009.
11. Wagenmakers E-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review. 2007; 14(5): 779-804.
12. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods. 2010; 1(2): 97-111. https://doi.org/10.1002/jrsm.12 PMID: 26061376
13. Ioannidis JP Why most discovered true associations are inflated. Epidemiology. 2008; 19(5): 640-8. https://doi.org/10.1097/EDE.0b013e31818131e7 PMID: 18633328
14. Lane DM, Dunlap WP. Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical & Statistical Psychology. 1978; 31: 107-12.
15. van Assen MALM, van Aert RCM, Wicherts JM. Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods. 2015; 20(3): 293-309. http://dx.doi.org/10.1037/met0000025. PMID: 25401773
16. Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester, UK: Wiley; 2005.
17. Neuliep JW, Crandall R. Everyone was wrong - There are lots of replications out there. Journal of Social Behavior and Personality. 1993; 8(6): 1-8.
18. van Aert RCM, Wicherts JM, van Assen MALM. Conducting meta-analyses on p-values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science. 2016; 11(5): 713-29. https://doi.org/10.1177/1745691616650874 PMID: 27694466
19. Simonsohn U, Nelson LD, Simmons JP. P-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science. 2014; 9(6): 666-81. https://doi.org/10.1177/1745691614553988 PMID: 26186117
20. McShane BB, Böckenholt U, Hansen KT. Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science. 2016; 11 (5): 730-49. https://doi.org/10.1177/1745691616662243 PMID: 27694467
21. van Aert RCM, van Assen MALM. Examining reproducibility in psychology: A hybrid method for statistically combining a biased original study and replication. Manuscript submitted for publication 2016.
22. Etz A, Vandekerckhove J. A Bayesian perspective on the Reproducibility Project: Psychology. PLoS ONE. 2016; 11(2).
23. Fisher RA On the 'probable error' of a coefficient of correlation deduced from a small sample. Metron. 1921; 1: 3-32.
24. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale NJ: Lawrence Erlbaum Associates; 1988.
25. Raudenbush SW Analyzing effect sizes: Random-effects models. In: Cooper H, Hedges LV, Valentine JC, editors. The Handbook of Research Synthesis and Meta-Analysis. New York: Russell Sage Foundation; 2009. p. 295-315.
26. Kass RE, Raftery AE. Bayes factors. Journal of the American Statistical Association. 1995; 90 (430): 791.
27. R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2017.
28. Bakker M, van Dijk A, Wicherts JM. The rules of the game called psychological science. Perspectives on Psychological Science. 2012; 7(6): 543-54. https://doi.org/10.1177/1745691612459060 PMID: 26168111
29. Cohen J. Things I have learned (so far). American Psychologist. 1990; 45(12): 1304-12.
30. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013; 14 (5): 365-76. https://doi.org/10.1038/nrn3475 PMID: 23571845
31. IntHout J, Ioannidis JP, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects metaanalysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology. 2014; 14.
32. Bruns SB, Ioannidis JP. P-curve and p-hacking in observational research. PLoS ONE. 2016; 11(2):: e0149144. https://doi.org/10.1371/journal.pone.0149144 PMID: 26886098
33. Ulrich R, Miller J. p-hacking by post hoc selection with multiple opportunities: Detectability by skewness test?: Comment on Simonsohn, Nelson, and Simmons (2014). Journal of Experimental Psychology: General. 2015; 144(6): 1137-45.