Bayesian mixture modeling of significant p values: A meta-analytic method to estimate the degree of contamination from H0

Journal of Experimental Psychology: General

Volumn 146, Issue 9, 2017, Pages 1223-1233

  Gronau, Quentin Frederik ^a   Duizer, Monique ^a   Bakker, Marjan ^b   Wagenmakers, Eric Jan ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b TILBURG UNIVERSITY   (Netherlands)

Author keywords

Bayesian mixture model; P curve; P hacking; Publication bias

Indexed keywords

BAYES THEOREM; HUMAN; PROBABILITY;

BAYES THEOREM; HUMANS; PROBABILITY;

EID: 85028430779     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/xge0000324     Document Type: Article

References (36)

1. Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, v., Birch, S., Birt, A. R., Zwaan, R. A. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 5, 556-578.
2. Barber, T. X. (1976). Pitfalls in human research: Ten pivotal points. New York, NY: Pergamon Press Inc.
3. Becker, B. J. (1991). Small-sample accuracy of approximate distributions of functions of observed probabilities from t tests. Journal of Educational Statistics, 16, 345-369.
4. Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex, 49, 609-610.
5. Chambers, C. D. (2015). Ten reasons why journals must review manuscripts before results are known. Addiction, 110, 10-11.
6. Chambers, C. D., Dienes, Z., McIntosh, R. D., Rotshtein, P., & Willmes, K. (2015). Registered reports: Realigning incentives in scientific publishing. Cortex, 66, A1-A2.
7. Cheung, I., Campbell, L., & LeBel, E. P. (2016). Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro, & Hannon (2002). Perspectives on Psychological Science, 11, 750-764.
8. Diebolt, J., & Robert, C. P. (1994). Estimation of finite mixture distributions through Bayesian sampling. Journal of the Royal Statistical Society, 56, 363-375.
9. Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P.,. Prenoveau, J. M. (2016). Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science, 11, 158-171.
10. Efron, B. (2012). Large-scale simultaneous hypothesis testing. Journal of the American Statistical Association, 99, 96-104.
11. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3-6.
12. Francis, G. (2013). Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology, 57, 153-169.
13. Frühwirth-Schnatter, S. (2006). Finite mixture and Markov switching models. New York, NY: Springer.
14. Gamerman, D., & Lopes, H. F. (2006). Markov Chain Monte Carlo: Stochastic simulation for Bayesian inference. Boca Raton, FL: Chapman & Hall/CRC.
15. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences (with discussion). Statistical Science, 7, 457-472.
16. Guan, M., & Vandekerckhove, J. (2016). A Bayesian approach to mitigation of publication bias. Psychonomic Bulletin & Review, 23, 74-86.
17. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532.
18. Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences of the United States of America, 110, 19313-19317.
19. Kahneman, D. (2011). Thinking, fast and slow. London, UK: Allen Lane.
20. Lee, M. D., & Wagenmakers, E.-J. (2013). Bayesian modeling for cognitive science: A practical course. New York, NY: Cambridge University Press.
21. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J.,. Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422-1425.
22. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
23. Pashler, H., & Wagenmakers, E.-J. (2012). Eds.' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528-530.
24. Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In K. Hornik, F. Leisch, & A. Zeileis (Eds.), Proceedings of the 3rd international workshop on distributed statistical computing, Vienna, Austria. Retrieved from https://www.rproject. org/conferences/DSC-2003/Proceedings/
25. Robert, C. P., & Casella, G. (1999). Monte Carlo statistical methods. New York, NY: Springer.
26. Rosenthal, R. (1979). An introduction to the file drawer problem. Psychological Bulletin, 86, 638-641.
27. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
28. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014a). p-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534-547.
29. Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014b). p-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666-681.
30. Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance- or vice versa. Journal of the American Statistical Association, 54, 30-34.
31. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
32. Tamhane, A. C., & Shi, J. (2009). Parametric mixture models for estimating the proportion of true null hypotheses and adaptive control of FDR. Lecture Notes-Monograph Series, 57, 304-325.
33. van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293-309.
34. Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R.,. Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11, 917-928.
35. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 627-633.
36. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298.