Limitations of PET-PEESE and Other Meta-Analysis Methods

Social Psychological and Personality Science

Volumn 8, Issue 5, 2017, Pages 581-591

  Stanley, T D ^a,b  

^a HENDRIX COLLEGE   (United States)

^b DEAKIN UNIVERSITY   (Australia)

Author keywords

meta analysis; PET PEESE; publication bias; random effects; weighted least squares

Indexed keywords

EID: 85028544800     PISSN: 19485506     EISSN: 19485514     Source Type: Journal    
DOI: 10.1177/1948550617693062     Document Type: Article

References (36)

1. Au J., Sheehan E., Tsai N., Duncan G. J., Buschkuehl M., Jaeggi S. M., (2015). Improving fluid intelligence with training on working memory: A meta-analysis. Psychological Bulletin Review, 22, 366–377.
2. Begg C. B., Berlin J. A., (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society. Series A, 151, 419–463.
3. Boggs T., Lasecki L., (2015). Reliable gains? Evidence for substantially underpowered designs in studies of working memory training transfer to fluid intelligence. Frontiers in Psychology. doi:10.3389/fpsyg.2014.01589
4. Bruns S. B., Ioannidis J. P. A., (2016). p-Curve and p-hacking in observational research. PLoS One, 11, e0149144. doi:10.1371/journal. pone.0149144
5. Carter E. C., Kofler L. M., Forster D. E., McCullough M. E., (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796–815.
6. Cohen J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, MI: Lawrence Erlbaum.
7. Copas J., Shi J. Q., (2000). Meta-analysis, funnel plots and sensitivity analysis. Biostatistics, 1, 247–262.
8. Egger M., Smith G. D., Schneider M., Minder C., (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
9. Fraley R. C., Vazire S., (2014). The n-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One, 9, e109019. doi:10.1371/journal. pone.0109019
10. Glass G. V., McGaw B., Smith M. L., (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
11. Greene W. E., (1990). Econometric analysis. New York, NY: Macmillan.
12. Hagger M. S., Chatzisarantis N. L. D., (2016). A multi-lab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573.
13. Hedges L. V., Olkin I., (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
14. Higgins J. P. T., Thompson S. G., (2002). Quantifying heterogeneity in meta-analysis. Statistics in Medicine, 21, 1539–1558.
15. Ioannidis J. P. A., Stanley T. D., Doucouliagos C. (H)., (in press). The power of bias in economics research. The Economic Journal. Also as SWP, Economics Series 2016/2, Deakin University, Australia. Retrieved July 21, 2016, from http://www.deakin.edu.au/__data/assets/pdf_file/0007/477763/2016_1.pdf.
16. Johnson N., Kotz S., (1970). Distributions in statistics: Continuous univariate distribution. New York, NY: Wiley.
17. Maxwell S. E., (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.
18. McShane B. B., Böckenholt U., Hansen K. T., (2016). Adjusting for publication bias in meta-analysis: An evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11, 730–749.
19. Moreno S. G., Sutton A. J., Ades A. E., Stanley T. D., Abrams K. R., Peters J. L., Cooper N. J., (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9, 2. Retrieved November 23, 2016, from http://www.biomedcentral.com/1471-2288/9/2
20. Moreno S. G., Sutton A. J., Thompson J. R., Ades A. E., Abrams K. R., Cooper N. J., (2012). A generalized weighting regression derived meta-analysis estimator robust to small-study effects and heterogeneity. Statistics in Medicine, 31, 1407–1417.
21. Moreno S. G., Sutton A. J., Turner E. H., Abrams K. R., Cooper N. J., Palmer T. M., Ades A. E., (2009). Novel methods to deal with publication biases: Secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, b298. Retrieved November 23, 2016, from http://dx.doi.org/10.1136/bmj.b2981
22. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716–aac4716. doi:10.1126/science.aac4716
23. Popper K., (1959). The logic of scientific discovery. New York, NY: Basic Books.
24. Richard F. D., Bond C. F., (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.
25. Rosenthal R., (1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641.
26. Schmidt F. L., Hunter J. E., (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: Sage.
27. Simonsohn U., Nelson L. D., Simmons J. P., (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547.
28. Stanley T. D., (2008). Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70, 103–127.
29. Stanley T. D., Doucouliagos H., (2012). Meta-regression analysis in economics and business. Oxford, England: Routledge.
30. Stanley T. D., Doucouliagos C. H., (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60–78.
31. Stanley T. D., Doucouliagos C. H., (2015). Neither fixed nor random: Weighted least squares meta-analysis. Statistics in Medicine, 34, 2116–2127.
32. Stanley T. D., Doucouliagos C. H., (2016). Neither fixed nor random: Weighted least squares meta-regression. Research Synthesis Methods. doi:10.1002/jrsm.1211
33. Stanley T. D., & Doucouliagos C. (H), Ioannidis J. P. A., (2017). Finding the power to reduce publication bias. Statistics in Medicine. Advance online publication. doi:10.1002/sim.7228
34. Stanley T. D., Jarrell S. B., Doucouliagos H., (2010). Could it be better to discard 90% of the data? A statistical paradox. The American Statistician, 64, 70–77.
35. Sterling T. D., (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association, 54, 30–34.
36. van Aert R. C. M., Wicherts J. M., van Assen M. A. L. M., (2016). Conducting meta-analyses based on p values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11, 713–729.