Testing for funnel plot asymmetry of standardized mean differences

Research Synthesis Methods

Volumn 10, Issue 1, 2019, Pages 57-71

  Pustejovsky, James E ^a   Rodgers, Melissa A ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

meta analysis; outcome reporting bias; publication bias; standardized mean difference

Indexed keywords

ANIMAL; COMPUTER LANGUAGE; COMPUTER SIMULATION; HUMAN; METHODOLOGY; MONTE CARLO METHOD; ODDS RATIO; PUBLISHING; REGRESSION ANALYSIS; REPRODUCIBILITY; SAMPLE SIZE; STANDARD; STATISTICAL ANALYSIS; STATISTICAL MODEL;

ANIMALS; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; LIKELIHOOD FUNCTIONS; MODELS, STATISTICAL; MONTE CARLO METHOD; ODDS RATIO; PROGRAMMING LANGUAGES; PUBLICATION BIAS; REFERENCE STANDARDS; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SAMPLE SIZE;

EID: 85063083008     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1332     Document Type: Article

References (65)

1. Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester: John Wiley & Sons; 2006.
2. Polanin JR, Tanner-Smith EE, Hennessy EA. Estimating the difference between published and unpublished effect sizes a meta-review. Rev Educ Res. 2016;86(1):207-236. http://rer.sagepub.com/content/86/1/207.short
3. Franco A, Malhotra N, Simonovits G. Underreporting in psychology experiments: evidence from a study registry. Social Psychol Personal Sci. 2016;7(1):8-12. https://doi.org/10.1177/1948550615598377
4. Ferguson CJ, Heene M. A vast graveyard of undead theories: publication bias and psychological science's aversion to the null. Perspect Psychol Sci. 2012;7(6):555-561. https://doi.org/10.1177/1745691612459059
5. Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials. 2007;4(3):245-253. https://doi.org/10.1177/1740774507079441
6. AllTrials. All trials registered. All results reported. AllTrials. http://www.alltrials.net/. Published 2014. Accessed January 19, 2018.
7. ClinicalTrials. ClinicalTrials.Gov. https://clinicaltrials.gov/. Published 2018. Accessed February 2, 2018.
8. Chambers CD. Registered Reports: A new publishing initiative at Cortex. Cortex. 2013;49(3):609-610. https://doi.org/10.1016/j.cortex.2012.12.016
9. Nosek BA, Lakens D. Registered reports: a method to increase the credibility of published results. Soc Psychol. 2014;45(3):137-141. https://doi.org/10.1027/1864-9335/a000192
10. Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR): Standards for the Conduct and Reporting of New Cochrane Intervention Reviews, Reporting of Protocols and the Planning, Conduct and Reporting of Updates. London: Cochrane; 2016.
11. Light RJ, Pillemer DB. Summing Up. Cambridge, MA:Harvard University Press; 1984.
12. Sterne JA. Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046-1055.
13. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088. https://doi.org/10.2307/2533446
14. Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Bmj. 1997;315(7109):629-634.
15. Duval S, Tweedie R. A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. J Am Stat Assoc. 2000;95(449):89-98. https://doi.org/10.2307/2669529
16. Stanley TD, Doucouliagos H. Meta-Regression Analysis in Economics and Business. London:Routledge; 2012.
17. Copas J, Shi JQ. A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res. 2001;10:251-265.
18. Hedges L, Vevea J. Selection method approaches. Publication Bias in Meta-Analysis: Prevention, Assessment, and Adjustments. Chichester, England: John Wiley & Sons; 2005:145-174.
19. Simonsohn U, Nelson LD, Simmons JP. P-curve and effect size: correcting for publication bias using only significant results. Perspect Psychol Sci. 2014;9(6):666-681. https://doi.org/10.1177/1745691614553988
20. Van Assen MALM, Van Aert RCM, Wicherts JM. Meta-analysis using effect size distributions of only statistically significant studies. Psychol Methods. 2015;20(3):293-309. https://doi.org/10.1037/met0000025
21. Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. https://doi.org/10.1136/bmj.d4002
22. Stuck AE, Rubenstein LZ, Wieland D. Bias in meta-analysis detected by a simple, graphical test. Asymmetry detected in funnel plot was probably due to true heterogeneity. BMJ: Br Med J. 1998;316(7129):469.
23. Tang J-L, Liu JL. Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol. 2000;53(5):477-484. https://doi.org/10.1016/S0895-4356(99)00204-8
24. Simonsohn U. The funnel plot is invalid because of this crazy assumption r(n,d)=0. http://datacolada.org/58. Published March 21, 2017. Accessed April 1, 2018.
25. Morey R. Asymmetric funnel plots without publication bias. http://bayesfactor.blogspot.com/2016/01/asymmetric-funnel-plots-without.html. Published January 9, 2016. Accessed April 1, 2018.
26. Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med. 2001;20(4):641-654. https://doi.org/10.1002/sim.698
27. Moreno SG, Sutton AJ, Ades A, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol. 2009;9(1):17. https://doi.org/10.1186/1471-2288-9-2
28. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. J Am Med Assoc. 2006;295(6):676-680.
29. Hunter JP, Saratzis A, Sutton AJ, Boucher RH, Sayers RD, Bown MJ. In meta-analyses of proportion studies, funnel plots were found to be an inaccurate method of assessing publication bias. J Clin Epidemiol. 2014;67(8):897-903. https://doi.org/10.1016/j.jclinepi.2014.03.003
30. Ismail Z, Elbayoumi H, Fischer CE, et al. Prevalence of depression in patients with mild cognitive impairment: a systematic review and meta-analysis. Jama Psychiatry. 2017;74(1):58-67.
31. Debray TP, Moons KG, Riley RD. Detecting small-study effects and funnel plot asymmetry in meta-analysis of survival data: a comparison of new and existing tests. Res Synth Methods. 2018;9(1):41-50. https://doi.org/10.1002/jrsm.1266
32. Zwetsloot PP, Van Der Naald M, Sena ES, et al. Standardized mean differences cause funnel plot distortion in publication bias assessments. eLife. 2017;6:1-20. https://doi.org/10.7554/eLife.24260
33. Hedges LV, Vevea JL. Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model. J Educ Behav Stat. 1996;21(4):299. https://doi.org/10.2307/1165338
34. McShane BB, Böckenholt U, Hansen KT. Adjusting for publication bias in meta-analysis an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci. 2016;11(5):730-749. http://pps.sagepub.com/content/11/5/730.short. Accessed November 1, 2016.
35. Carter E, Schönbrodt F, Gervais WM, Hilgard J. Correcting for bias in psychology: A comparison of meta-analytic methods. https://osf.io/preprints/psyarxiv/9h3nu/download?format=pdf; 2017.
36. Borenstein M. Effect sizes for continuous data. In: Cooper HM, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-Analysis. New York, NY: Russell Sage Foundation; 2009:221-236.
37. Feingold A. Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychol Methods. 2009;14(1):43-53. https://doi.org/10.1037/a0014699
38. Hedges LV. Effect sizes in cluster-randomized designs. J Educ Behav Stat. 2007;32(4):341-370. https://doi.org/10.3102/1076998606298043
39. Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. Orlando, FL: Academic Press; 1985.
40. Pigott T. Advances in Meta-Analysis. New York: Springer Science & Business Media; 2012.
41. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177-188.
42. Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Stat Med. 2003;22(17):2693-2710. https://doi.org/10.1002/sim.1482
43. Sidik K, Jonkman JN. Robust variance estimation for random effects meta-analysis. Comput Stat Data Anal. 2006;50(12):3681-3701. https://doi.org/10.1016/j.csda.2005.07.019
44. R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; https://www.R-project.org/; 2017.
45. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Software. 2010;36(3):1-48.
46. Coburn KM, Vevea JL. Weightr: Estimating weight-function models for publication bias. 2017. https://CRAN.R-project.org/package=weightr.
47. Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences: supplementary materials. 2018. https://osf.io/4bz7q/
48. Ahn S, Ames AJ, Myers ND. A review of meta-analyses in education: methodological strengths and weaknesses. Rev Educ Res; 82(4):436-476. https://doi.org/10.3102/0034654312458162
49. Fraley RC, Vazire S. The n-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS ONE. 2014;9(10):e109019. https://doi.org/10.1371/journal.pone.0109019
50. Carter EC, Kofler LM, Forster DE, McCullough ME. A series of meta-analytic tests of the depletion effect: self-control does not seem to rely on a limited resource. J Exp Psychol: Gen. 2015;144(4):796-815. https://doi.org/10.1037/xge0000083
51. Inzlicht M, Gervais W, Berkman E. Bias-correction techniques alone cannot determine whether ego depletion is different from zero: Commentary on Carter, Kofler, Forster, & McCullough. 2015. Available at SSRN: http://dx.doi.org/10.2139/ssrn.2659409
52. Park S, Beretvas SN. Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model. Behav Res Methods. 2018. https://doi.org/10.3758/s13428-018-1156-y
53. Stanley TD, Carter EC, Doucouliagos H. What Meta-Analyses Reveal About the Replicability of Psychological Research. Psychol Bull. 2018;144(12):1325-1346. https://doi.org/10.1037/bul0000169
54. Van Erp S, Verhagen J, Grasman RPPP, Wagenmakers E-J. Estimates of between-study heterogeneity for 705 meta-analyses reported in psychological bulletin from 1990–2013. J Open Psychol Data. 2017;5(1):1-5. https://doi.org/10.5334/jopd.33
55. Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika. 1995;60(3):419-435. https://doi.org/10.1007/BF02294384
56. Coburn KM, Vevea JL. Publication bias as a function of study characteristics. Psychol Methods. 2015;20(3):310-330. https://doi.org/10.1037/met0000046
57. Citkowicz M, Vevea JL. A parsimonious weight function for modeling publication bias. Psychol Methods. 2017;22(1):28-41. https://doi.org/10.1037/met0000119
58. Rufibach K. Selection models with monotone weight functions in meta analysis. Biochem J. 2011;53(4):689-704. https://doi.org/10.1002/bimj.201000240
59. Du H, Liu F, Wang L. A Bayesian “fill-in” method for correcting for publication bias in meta-analysis. Psychol Methods. 2017;22(4):799.
60. Becker BJ. Multivariate meta-analysis. Handbook of Applied Multivariate Statistics and Mathematical Modeling. San Diego, CA: Academic Press; 2000:499-525. https://doi.org/10.1016/B978-012691360-6/50018-5
61. Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Res Synth Methods. 2010;1(1):39-65. https://doi.org/10.1002/jrsm.5
62. Jackson D, Riley RD, White IR. Multivariate meta-analysis: potential and promise. Stat Med. 2011;30:2481-2498. https://doi.org/10.1002/sim.4172
63. Van den Noortgate W, López-López JA, Marín-Martínez F, Sánchez-Meca J. Three-level meta-analysis of dependent effect sizes. Behav Res Methods. 2013;45(2):576-594. https://doi.org/10.3758/s13428-012-0261-6
64. Chen Y, Hong C, Riley RD. An alternative pseudolikelihood method for multivariate random-effects meta-analysis. Stat Med. 2015;34(3). https://doi.org/10.1002/sim.6350
65. Friese M, Frankenbach J, Job V, Loschelder DD. Does self-control training improve self-control? A meta-analysis. Perspect Psychol Sci. 2017;12(6):1077-1099. https://doi.org/10.1177/1745691617697076