Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes

Perspectives on Psychological Science

Volumn 11, Issue 5, 2016, Pages 730-749

  McShane, Blakeley B ^a   Böckenholt, Ulf ^a   Hansen, Karsten T ^b  

^a NORTHWESTERN UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

effect size; meta analysis; p curve; p uniform; selection methods

Indexed keywords

COMPUTER SIMULATION; EVALUATION STUDY; HUMAN; META ANALYSIS (TOPIC); PUBLISHING; STATISTICAL ANALYSIS; STATISTICAL MODEL;

COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; PUBLICATION BIAS;

EID: 84989253877     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691616662243     Document Type: Article

References (51)

1. Becker B. J., (2005). Failsafe n or file-drawer number. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111–125). Chichester, England: John Wiley & Sons.
2. Begg C. B., Mazumdar M., (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.
3. Bickel P., Doksum K., (2007). Mathematical statistics: Basic ideas and selected topics (Vol. 1., 2nd ed.). Upper Saddle River, NJ: Pearson Prentice Hall.
4. Boyle R., (1965). The sceptical chymist. London, England: Dawsons of Pall Mall. (Original work published 1661)
5. Casella G., Berger R. L., (2002). Statistical inference. Pacific Grove, CA: Duxbury.
6. Coburn K. M., Vevea J. L., (2016). Weightr: Estimating weight-function models for publication bias in r. R package version 1.0.0. Retrieved from https://CRAN.R-project.org/package=weightr
7. Cohen J., (1994). The earth is round (p <.05). American Psychologist, 49, 997–1003.
8. Copas J. B., (1999). What works? Selectivity models and meta-analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162, 95–109.
9. Copas J. B., (2013). A likelihood-based sensitivity analysis for publication bias in meta-analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 62, 47–66.
10. Copas J. B., Li H., (1997). Inference for non-random samples. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 59, 55–95.
11. Copas J. B., Shi J. Q., (2001). A sensitivity analysis for publication bias in systematic reviews. Statistical Methods in Social Research, 10, 251–265.
12. Dear K. B., Begg C. B., (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237–245.
13. Dickersin K., (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11–33). Chichester, England: John Wiley & Sons.
14. Duval S., (2005). The trim and fill method. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 127–144). Chichester, England: John Wiley & Sons.
15. Duval S., Tweedie R., (2000a). A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89–98.
16. Duval S., Tweedie R., (2000b). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463.
17. Editors. (1909). The reporting of unsuccessful cases [Editorial]. The Boston Medical and Surgical Journal, 161, 263–264.
18. Egger M., Smith G. D., Schneider M., Minder C., (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629–634.
19. Fanelli D., (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5(4), e10068. doi:10.1371/journal.pone.0010068
20. Fanelli D., (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904.
21. Ferriar J., Simmons W., (1792). Medical histories and reflection. London, England: Cadell and Davies.
22. Fisher Ronald A, (1938). Presidential address, First Indian Statistical Conference. Sankhya, 4, 14–17.
23. Gelman A., (2015). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management, 41, 632–643.
24. Hall M. B., (1959). In defense of experimental essays. In Boyle R., Hall M. B., (Eds.), Robert Boyle on natural philosophy: An essay, with selections from his writings (pp. 119–131). Bloomington, IN: Indiana University Press.
25. Hedges L. V., (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61–85.
26. Hedges L. V., (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255.
27. Hedges L. V., Vevea J. L., (2005). Selection method approaches. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 145–174). Chichester, England: John Wiley & Sons.
28. Iyengar S., Greenhouse J. B., (1988). Selection models and the file drawer problem. Statistical Science, 3, 109–117.
29. Jensen J. L. W. V. (1906). Sur les fonctions convexes et les inégalités entre les valeurs moyennes [On convex functions and inequality between the average values]. Acta Mathematica, 30, 175–193.
30. Jin Z.-C., Zhou X.-H., He J., (2015). Statistical methods for dealing with publication bias in meta-analysis. Statistics in Medicine, 34, 343–360.
31. Klein R. A., Ratliff K., Nosek B. A., Vianello M., Pilati R., Devos T... Kappes H., (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142–152.
32. Lane D. M., Dunlap W. P., (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107–112.
33. McShane B. B., Böckenholt U., (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives on Psychological Science, 9, 612–625.
34. McShane B. B., Böckenholt U., (2016). Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach. Psychological Methods, 21, 47–60.
35. McShane B. B., Gal D., (2016). Blinding us to the obvious? The effect of statistical training on the evaluation of evidence. Management Science, 62, 1707–1718.
36. Pigott T., (2012). Advances in meta-analysis. New York, NY: Springer.
37. R Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
38. Rosenthal R., (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
39. Rothstein H. R., Sutton A. J., Borenstein M., (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester, England: John Wiley & Sons.
40. Schmidt F. L., Hunter J. E., (2014). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage Publications.
41. Simonsohn U., Nelson L. D., Simmons J. P., (2014). p-curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666–681.
42. Stanley T., Doucouliagos H., (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60–78.
43. Sterling T. D., (1959). Publication decisions and their possible effects on inferences drawn from tests of significance—Or vice versa. Journal of the American Statistical Association, 54, 30–34.
44. Sterling T. D., Rosenbaum W., Weinkam J., (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108–112.
45. Sterne J. A. C., Becker B. J., Egger M., (2005). The funnel plot. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 75–98). Chichester, England: John Wiley & Sons.
46. Sterne J. A. C., Egger M., (2005). Regression methods to detect publication and other bias in meta-analysis. In Rothstein H. R., Sutton A. J., Borenstein M., (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 99–110). Chichester, England: John Wiley & Sons.
47. Tukey J. W., (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100–116.
48. van Aert R. C. M., Wicherts J. M., van Assen M. A. L. M. (2016). Conducting meta-analyses based on p-values: Reservations and recommendations for applying p-uniform and p-curve. Perspectives on Psychological Science, 11, 713–729.
49. van Assen M. A., van Aert R., Wicherts J. M. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293–309.
50. Vevea J. L., Hedges L. V., (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60, 419–435.
51. Vevea J. L., Woods C. M., (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428–443.