Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis

International Journal of Social Research Methodology

Volumn 20, Issue 6, 2017, Pages 559-572

  Moeyaert, Mariola ^a   Ugille, Maaike ^b   Natasha Beretvas, S ^c   Ferron, John ^d   Bunuan, Rommel ^c   Van den Noortgate, Wim ^b  

^a STATE UNIVERSITY OF NEW YORK   (United States)

^b UNIVERSITY OF LEUVEN   (Belgium)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^d UNIVERSITY OF SOUTH FLORIDA   (United States)

Author keywords

averaging effects; dependent effect sizes; Meta analysis; multilevel meta analysis; robust variance estimation

Indexed keywords

EID: 84994904809     PISSN: 13645579     EISSN: 14645300     Source Type: Journal    
DOI: 10.1080/13645579.2016.1252189     Document Type: Article

References (33)

1. Ahn, S., Ames, A. J., & Meyers, N. D., (2012). A review of meta-analysis in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436–476.10.3102/0034654312458162
2. Becker, B. J., (2000). Multivariate meta-analysis. In H. E. A., Tinsley & E. D., Brown (Eds.), Handbook of applied multivariate statistics and mathematcial modeling (pp. 499–525). Orlando, FL: Academic Press.10.1016/B978-012691360-6/50018-5
3. Berkey, C. S., Anderson, J. J., & Hoaglin, D. C., (1996). Multiple-outcome meta-analysis of clinical trials. Statistics in Medicine, 15, 537–557.10.1002/(ISSN)1097-0258
4. Borenstein, M., Hedges, L., Higgens, J., & Rothstein, H., (2009). Introduction to meta-analysis. West Sussex: Wiley.10.1002/9780470743386
5. Cheung, M., (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19, 211–229.10.1037/a0032968
6. Cheung, S., & Chan, D., (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780–791.10.1037/0021-9010.89.5.780
7. Cohen, J., (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
8. Cooper, H., (2010). Research synthesis and meta-analysis (4th ed.). London: Sage.
9. D’Agostino, J., & Murphy, J., (2004). A meta-analysis of reading recovery in United States schools. Educational Evaluation and Policy Analysis, 26, 23–28.10.3102/01623737026001023
10. Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C. K., Cable, A., & Tackett, K. K., (2009). A synthesis of reading interventions and effects on reading outcomes for older struggling readers. Review of Educational Research, 79, 262–300.10.3102/0034654308325998
11. Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M., (2001). Systematic phonics instruction helps students learn to read: evidence from the national reading panel’s meta-analysis. Review of Educational Research, 71, 393–447.10.3102/00346543071003393
12. Fisher, Z., & Tipton, E., (2014). Robumeta: Robust variance meta-regression. Retrieved from http://cran.r-project.org/web/packages/robumeta/robumeta.pdf
13. Gleser, L. J., & Olkin, I., (2009). Stochastically dependent effect sizes. In H., Cooper, L. V., Hedges, & J. C., Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York, NY: Russell Sage Foundation.
14. Hedges, L. V., (2007). Correcting a significance test for clustering. Journal of Educational and Behavioral Statistics, 32, 151–179.10.3102/1076998606298040
15. Hedges, L. V., & Olkin, I., (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
16. Hedges, L. V., Tipton, E., & Johnson, M. C., (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.10.1002/jrsm.v1:1
17. Hoogland, J. J., & Boomsma, A., (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26, 329–367.10.1177/0049124198026003003
18. Hox, J., (2002). Multilevel analysis. MahWah, NJ: Erlbaum.
19. Jenson, W. R., Clark, E., Kircher, J. C., & Kristjansson, S. D., (2007). Statistical reform: Evidence-based practice, meta-analyses, and single subject designs. Psychology in the Schools, 44, 483–493.10.1002/(ISSN)1520-6807
20. Kalaian, H. A., & Raudenbush, S. W., (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227–235.10.1037/1082-989X.1.3.227
21. Konstantopoulos, S., (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2, 61–76.10.1002/jrsm.v2.1
22. Lipsey, M., & Wilson, D., (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
23. Manolov, R., & Solanas, A., (2013). A comparison of mean phase difference and generalized least squares for analyzing single-case data. Journal of School Psychology, 51, 201–215.10.1016/j.jsp.2012.12.005
24. Matt, G. E., & Cook, T. D., (2009). Threats to the validity of research syntheses. In H. M., Cooper & L. V., Hedges (Eds.), Handbook of research synthesis (2nd ed., pp. 537–560). New York, NY: Russell Sage.
25. Parker, R. I., & Brossart, D. F., (2003). Evaluating single-case research data: A comparison of seven statistical methods. Behavior Therapy, 34, 189–211.10.1016/S0005-7894(03)80013-8
26. Raudenbush, S. W., Becker, B. J., & Kalaian, H. A., (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120.10.1037/0033-2909.103.1.111
27. Raudenbush, S. W., & Bryk, A. S., (2002). Hierarchical linear models: Applications and data analysis methods, Vol. 2nd. London: Sage.
28. SAS Institute Inc. (2011–2014). SAS® 9.3 Software [Computer software].
29. Tanner-Smith, E., & Tipton, E., (2014). Robust variance estimation with dependent effect sizes: Practical considerations and a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30.10.1002/jrsm.v5.1
30. Tipton, E., (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20, 375–393.10.1037/met0000011
31. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J., (2013). Three level meta-analysis of dependent effect sizes. Behavior Research Methods, 45, 576–594.10.3758/s13428-012-0261-6
32. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J., (2015). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294.
33. Van den Noortgate, W., & Onghena, P., (2003). Multilevel meta-analysis: A comparison with traditional meta-analytic procedures. Educational and Psychological Measurement, 63, 765–790.10.1177/0013164403251027