Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation

Research Synthesis Methods

Volumn 8, Issue 4, 2017, Pages 435-450

  López López, José Antonio ^a   Van den Noortgate, Wim ^b   Tanner Smith, Emily E ^c   Wilson, Sandra Jo ^c   Lipsey, Mark W ^c  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

^b UNIVERSITY OF LEUVEN   (Belgium)

^c VANDERBILT UNIVERSITY   (United States)

Author keywords

dependency; meta analysis; meta regression; robust variance estimation; three level model

Indexed keywords

ALGORITHM; COMPUTER SIMULATION; HUMAN; META ANALYSIS (TOPIC); MONTE CARLO METHOD; REGRESSION ANALYSIS; REPRODUCIBILITY; STATISTICAL ANALYSIS; STATISTICAL BIAS; STATISTICAL MODEL;

ALGORITHMS; BIAS; COMPUTER SIMULATION; DATA INTERPRETATION, STATISTICAL; HUMANS; LIKELIHOOD FUNCTIONS; META-ANALYSIS AS TOPIC; MODELS, STATISTICAL; MONTE CARLO METHOD; REGRESSION ANALYSIS; REPRODUCIBILITY OF RESULTS;

EID: 85019698524     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1245     Document Type: Article

References (44)

1. Hedges LV, Olkin I. Statistical Methods for Meta-analysis. Orlando, FL: Academic Press; 1985.
2. Lipsey MW, Wilson DB. Practical Meta-analysis. Thousand Oaks, CA: Sage; 2001.
3. Sánchez-Meca J, Marín-Martínez F. Meta-analysis. In: Peterson P, Baker E, McGaw B, eds. International Encyclopedia of Education. 3rd ed. Vol.7 Oxford: Elsevier; 2010:274-282.
4. Lipsey MW. Identifying interesting variables and analysis opportunities. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:147-158.
5. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI, Health Outcomes, Policy, and Economics (HOPE) Collaborative Group. Understanding heterogeneity in meta-analysis: the role of meta-regression. Int J Clin Pract. 2009;63:1426-1434.
6. Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002;21:1559-1573.
7. Stevens JR, Taylor AM. Hierarchical dependence in meta-analysis. J Educ Behav Stat. 2009;34:46-73.
8. Ahn S, Myers ND, Jin Y. Use of the estimated intraclass correlation for correcting differences in effect size by level. Behav Res Meth. 2012;44:490-502.
9. Becker BJ. Multivariate meta-analysis. In: Tinsley HEA, Brown SD, eds. Handbook of Applied Multivariate Statistics and Mathematical Modeling. San Diego, CA: Academic Press; 2000:499-525.
10. Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates. Res Synth Meth. 2010;1:39-65.
11. Van den Noortgate W, López-López JA, Marín-Martínez F, Sánchez-Meca J. Three-level meta-analysis of dependent effect sizes. Behav Res Meth. 2013;45:576-594.
12. Van den Noortgate W, López-López JA, Marín-Martínez F, Sánchez-Meca J. Meta-analysis of multiple outcomes: a multilevel approach. Behav Res Meth. 2015;47:1274-1294.
13. Gleser LJ, Olkin I. Stochastically dependent effect sizes: Random-effects models. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:357-376.
14. Tipton E. Robust variance estimation in meta-regression with binary dependent effects. Res Synth Meth. 2013;4:169-187.
15. Marín-Martínez F, Sánchez-Meca J. Averaging dependent effect sizes in meta-analysis: a cautionary note about procedures. Span J Psychol. 1999;2:32-38.
16. Rosenthal R, Rubin DB. Meta-analytic procedures for combining studies with multiple effect sizes. Psychol Bull. 1986;99:400-406.
17. Jackson D, Riley R, White IR. Multivariate meta-analysis: potential and promise. Stat Med. 2011;30:2481-2498.
18. Cheung MWL. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol Meth. 2014;19:211-229.
19. Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Res Synth Meth. 2011;2:61-76.
20. Hunter JE, Schmidt FL. Methods of Meta-analysis: Correcting Errors and Bias in Research Findings. 2nd ed. Newbury Park, CA: Sage; 2004.
21. DerSimonian R, Laird N. Meta-analysis of clinical trials. Clin Contr Trials. 1986;7:177-188.
22. Raudenbush SW. Analyzing effect sizes: Random-effects models. In: Cooper H, Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-analysis. 2nd ed. New York: Russell Sage Foundation; 2009:295-315.
23. Sidik K, Jonkman JN. A note on variance estimation in random effects meta-regression. J Biopharm Stat. 2005;15:823-838.
24. Viechtbauer W, López-López JA, Sánchez-Meca J, Marín-Martínez F. A comparison of procedures to test for moderators in mixed-effects meta-regression models. Psychol Meth. 2015;20:360-374.
25. Cribari-Neto F, Da Silva WB. A new heteroskedasticity-consistent covariance matrix estimator for the linear regression model. Adv Stat Anal. 2011;95:129-146.
26. MacKinnon JG, White H. Some heteroskedastity-consistent covariance matrix estimators with improved finite sample properties. J Econometrics. 1985;29:305-325.
27. Tipton E. Small sample adjustments for robust variance estimation with meta-regression. Psychol Meth. 2015;20:375-393.
28. Bell RM, McCaffrey DF. Bias reduction in standard errors for linear regression with multi-stage samples. Surv Meth. 2002;28:169-181.
29. McCaffrey DF, Bell RM, Botts CH 2001. Generalizations of biased reduced linearization. Proceedings of the Annual Meeting of the American Statistical Association, August 5–9, 2001.
30. Fisher Z, Tipton E 2014. “robumeta: robust variance meta-regression. R package version 1.6”. Available at: http://cran.r-project.org/web/packages/robumeta/index.html (Accessed 1 September 2016)
31. Tanner-Smith EE, Tipton E. Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and SPSS. Res Synth Meth. 2014;5:13-30.
32. Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2nd ed. London: Sage Publications; 2002.
33. Van den Noortgate W, Onghena P. Multilevel meta-analysis: a comparison with traditional meta-analytical procedures. Educ Psychol Meas. 2003;63:765-790.
34. Beretvas SN, Pastor DA. Using mixed-effects models in reliability generalization studies. Educ Psychol Meas. 2003;63:75-95.
35. Snijders T, Bosker R. Multilevel Modeling: An Introduction to Basic and Advanced Multilevel Modeling. London: Sage Publications; 1999.
36. Wilson SJ, Lipsey MW. School-based interventions for aggressive and disruptive behavior: update of a meta-analysis. Am J Prev Med. 2007;33:S130-S143.
37. Wilson SJ, Lipsey MW, Derzon JH. The effects of school-based intervention programs on aggressive and disruptive behavior: a meta-analysis. J Consult Clin Psychol. 2003;2003(71):136-149.
38. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Software. 2010;36:1-48.
39. Cheung MWL. metaSEM: an R package for meta-analysis using structural equation modeling. Front Psych. 2015;5:1521
40. Boker S, Neale M, Maes H, et al. OpenMx: an open source extended strucutral equation modeling framework. Psychometrika. 2011;76:306-317.
41. Sánchez-Meca J, Marín-Martínez F. Testing continuous moderators in meta-analysis: a comparison of procedures. Br J Math Stat Psychol. 1998;51:311-326.
42. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Meth. 2010;1:97-111.
43. Deeks JJ, Higgins JPT, Altman DG. Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S, eds. Cochrane Handbook of Systematic Reviews of Interventions. Chichester: John Wiley & Sons; 2008:243-296.
44. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Eribaum; 1988.