Dealing with effect size multiplicity in systematic reviews and meta-analyses

Research Synthesis Methods

Volumn 9, Issue 3, 2018, Pages 336-351

  López López, José A ^a   Page, Matthew J ^a,b   Lipsey, Mark W ^c   Higgins, Julian P T ^a  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b MONASH UNIVERSITY   (Australia)

^c VANDERBILT UNIVERSITY   (United States)

Author keywords

dependency; effect size; meta analysis; multiplicity; systematic review

Indexed keywords

EID: 85051034589     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1310     Document Type: Article

References (58)

1. Page MJ, McKenzie JE, Chau M, Green SE, Forbes A. Methods to select results to include in meta-analyses deserve more consideration in systematic reviews. J Clin Epidemiol. 2015;68(11):1282-1291.
2. Wilson SJ, Polanin JR, Lipsey MW. Fitting meta-analytic structural equation models with complex datasets. Res Syn Meth. 2016;7(2):121-139.
3. Stevens JR, Taylor AM. Hierarchical dependence in meta-analysis. J Educ Behav Stat. 2009;34(1):46-73.
4. Littell JH, Corcoran J, Pillai V. Systematic Reviews and Meta-analysis. New York: Oxford University Press; 2008.
5. Hedges LV. Statistical considerations. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:37-47.
6. Van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med. 2002;21(4):589-624.
7. Van den Noortgate W, López-López JA, Marín-Martínez F, Sánchez-Meca J. Three-level meta-analysis of dependent effect sizes. Behav Res Methods. 2013;45(2):576-594.
8. Van den Noortgate W, López-López JA, Marín-Martínez F, Sánchez-Meca J. Meta-analysis of multiple outcomes: a multilevel approach. Behav Res Methods. 2015;47(4):1274-1294.
9. Tendal B, Nüesch E, Higgins JPT, Jüni P, Gøtzsche PC. Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study. BMJ. 2011;343:d4829.
10. Bender R, Bunce C, Clarke M, et al. Attention should be given to multiplicity issues in systematic reviews. J Clin Epidemiol. 2008;61(9):857-865.
11. Davey J, Turner RM, Clarke MJ, Higgins JPT. Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Med Res Methodol. 2011;11(1):160.
12. Sadeghirad B, Siemieniuk RAC, Brignardello-Petersen R, et al. Corticosteroids for treatment of sore throat: systematic review and meta-analysis of randomised trials. BMJ. 2017;358:j3887.
13. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI. Understanding heterogeneity in meta-analysis: the role of meta-regression. Int J Clin Pract. 2009;63(10):1426-1434.
14. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009;172(1):137-159.
15. Viechtbauer W. Analysis of moderator effects in meta-analysis. In: Osborne J, ed. Best Practices in Quantitative Methods. Thousand Oaks, CA: Sage; 2008:471-487.
16. Wilson SJ, Lipsey MW, Derzon JH. The effects of school-based intervention programs on aggressive and disruptive behavior: a meta-analysis. J Consult Clin Psychol. 2003;71(1):136-149.
17. Sánchez-Meca J, Rosa-Alcázar AI, Marín-Martínez F, Gómez-Conesa A. Psychological treatment of panic disorder with or without agoraphobia: a meta-analysis. Clin Psychol Rev. 2010;30(1):37-50.
18. Hofmann SG, Smits JA. Cognitive-behavioral therapy for adult anxiety disorders: a meta-analysis of randomized placebo-controlled trials. J Clin Psychiatry. 2008;69(4):621-632.
19. Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. BMJ. 2009;339:b2535.
20. Hedges LV, Olkin I. Statistical Methods for Meta-analysis. Orlando, FL: Academic Press; 1985.
21. Rosenthal R, Rubin DB. Meta-analytic procedures for combining studies with multiple effect sizes. Psychol Bull. 1986;99(3):400-406.
22. Marín-Martínez F, Sánchez-Meca J. Averaging dependent effect sizes in meta-analysis: a cautionary note about procedures. Span J Psychol. 1999;2(1):32-38.
23. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-analysis. Chichester, UK: Wiley; 2009.
24. Hedges LV. Meta-analysis. In: Rao CR, Sinharay S, eds. Handbook of Statistics. Vol.26 Amsterdam: Elsevier; 2007:919-953.
25. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: Wiley-Blackwell; 2011.
26. Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. In: Agency for Healthcare Research and Quality; 2008.
27. Boers M, Kirwan JR, Wells G, et al. Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67(7):745-753.
28. Christensen R, Maxwell LJ, Jüni P, et al. Consensus on the need for a hierarchical list of patient-reported pain outcomes for metaanalyses of knee osteoarthritis trials: an OMERACT objective. J Rheumatol. 2015;42(10):1971-1975.
29. Prinsen CA, Vohra S, Rose MR, et al. Core Outcome Measures in Effectiveness Trials (COMET) initiative: protocol for an international Delphi study to achieve consensus on how to select outcome measurement instruments for outcomes included in a ‘core outcome set’. Trials. 2014;15(1):247.
30. Mayo-Wilson E, Li T, Fusco N, et al. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. J Clin Epidemiol. 2017.
31. Higgins JPT, Soares-Weiser K, López-López JA, et al. Association of BCG, DTP, and measles containing vaccines with childhood mortality: systematic review. BMJ. 2016;355:i5170.
32. Becker BJ. Multivariate meta-analysis. In: Tinsley HEA, Brown SD, eds. Handbook of Applied Multivariate Statistics and Mathematical Modeling. San Diego, CA: Academic Press; 2000:499-525.
33. Riley RD, Price MJ, Jackson D, et al. Multivariate meta-analysis using individual participant data. Res Syn Meth. 2015;6(2):157-174.
34. Gleser LJ, Olkin I. Stochastically dependent effect sizes. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:357-376.
35. Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J. Statistical multiplicity in systematic reviews of anaesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One. 2011;6(12):e28422.
36. Kirkham JJ, Riley RD, Williamson PR. A multivariate meta-analysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31(20):2179-2195.
37. Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Syn Meth. 2012;3(2):80-97.
38. Ades AE, Lu G, Dias S, Mayo-Wilson E, Kounali D. Simultaneous synthesis of treatment effects and mapping to a common scale: an alternative to standardisation. Res Syn Meth. 2015;6(1):96-107.
39. Riley R, Thompson J, Abrams K. An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics. 2008;9:172-186.
40. Wei Y, Higgins JPT. Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Stat Med. 2013;32(7):1191-1205.
41. White IR. Multivariate random-effects meta-regression: updates to mvmeta. Stata J. 2011;11(2):255.
42. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1-48.
43. Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Res Syn Meth. 2011;2(1):61-76.
44. Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2nd ed. Thousand Oaks, CA: Sage; 2002.
45. Cheung MWL. metaSEM: an R package for meta-analysis using structural equation modeling. Front Psych. 2015;5:1521.
46. Bauer DJ. Estimating multilevel linear models as structural equation models. J Educ Behav Stat. 2003;28(2):135-167.
47. Cheung MWL, Chan W. Meta-analytic structural equation modeling: a two-stage approach. Psychol Methods. 2005;10(1):40-64.
48. Cheung MWL. Fixed-and random-effects meta-analytic structural equation modeling: examples and analyses in R. Behav Res Methods. 2014;46(1):29-40.
49. Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Res Syn Meth. 2010;1(1):39-65.
50. Huber P. The behavior of maximum-likelihood estimates under nonstandard conditions. In: LeCam LM, Neyman J, eds. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, CA: University of California Press; 1967:221-233.
51. White H. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica. 1980;48(4):817-838.
52. Tipton E. Small sample adjustments for robust variance estimation with meta-regression. Psychol Methods. 2015;20(3):375-393.
53. Tipton E, Pustejovsky JE. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J Educ Behav Stat. 2015;40(6):604-634.
54. Tanner-Smith EE, Tipton E. Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Res Syn Meth. 2014;5(1):13-30.
55. Greenland S, Longnecker MP. Methods for trend estimation from summarized dose-response data, with applications to meta-analysis. Am J Epidemiol. 1992;135(11):1301-1309.
56. Polanin JR, Pigott TD. The use of meta-analytic statistical significance testing. Res Syn Meth. 2015;6(1):63-73.
57. López-López JA, Van den Noortgate W, Tanner-Smith EE, Wilson SJ, Lipsey MW. Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: a Monte Carlo simulation. Res Syn Meth. 2017;8(4):435-450.
58. Lipsey MW. Identifying interesting variables and analysis opportunities. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:147-158.