Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model

Behavior Research Methods

Volumn 51, Issue 1, 2019, Pages 152-171

  Park, Sunyoung ^a   Beretvas, S Natasha ^b  

^a CALIFORNIA LUTHERAN UNIVERSITY   (United States)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Multilevel meta analysis; Multiple outcome dependency; Robust variance estimation; Simulation study

Indexed keywords

ARTICLE; CONTROLLED STUDY; EFFECT SIZE; HUMAN; META ANALYSIS; SAMPLE SIZE; SIMULATION; MULTILEVEL ANALYSIS; OUTCOME ASSESSMENT; PROCEDURES; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; MULTILEVEL ANALYSIS; OUTCOME ASSESSMENT (HEALTH CARE); SAMPLE SIZE;

EID: 85056298873     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-018-1156-y     Document Type: Article

References (42)

1. Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., & Persson, T. (2015). Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research, 85, 275–314.
2. Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015). Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review. Clinical Psychology Review, 42, 47–61.
3. Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Orlando, FL: Academic Press.
4. Becker, B. J., Hedges, L. V., & Pigott, T. D. (2004). Campbell Collaboration statistical analysis policy brief (Campbell Collaboration resource document). Available from http://www.campbellcollaboration.org/ECG/policy_stat.asp
5. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken, NJ: Wiley.
6. Cheung, M. W.-L. (2015). Meta-analysis: A structural equation modeling approach. Chichester, UK: Wiley.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
8. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.
9. Fisher, Z., & Tipton, E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Working paper, Department of Statistics, Northwestern University, Evanston, IL.
10. Geeraert, L., Van den Noortgate, W., Grietens, H., & Onghena, P. (2004). The effects of early prevention programs for families with young children at risk for physical child abuse and neglect: A meta-analysis. Child Maltreatment, 9, 277–291.
11. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
12. Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 357–376). New York, NY: Russell Sage Foundation.
13. Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128.
14. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
15. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation of meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.
16. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. doi:10.1037/1082-989X.3.4.486
17. Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. Sociological Methods and Research, 26, 329–367.
18. Hox, J. J., & Maas, C. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8, 157–174.
19. Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). New York, NY, Routledge.
20. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
21. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997.
22. Kisamore, J. L., & Brannick, M. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35–53.
23. Kreft, I. G. G., & De Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.
24. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-2241.1.3.86
25. Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 538–557). New York, NY: Sage.
26. McNeish, D. M., & Stapleton, L. M. (2016a). The effect of small sample size on two-level model estimates: A review and illustration. Educational Psychology Review, 28.
27. McNeish, D. M., & Stapleton, L. M. (2016b). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51, 495–518.
28. McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52, 200–215.
29. Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017) Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis, International Journal of Social Research Methodology, 20, 559–572.
30. National Research Council. (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy of Science Press.
31. Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 301–322). New York, NY: Russell Sage Foundation.
32. Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120. doi:10.1037/0033-2909.103.1.111
33. Schmidt, F. L., Oh, I., & Hayes, T. L. (2009). Fixed- versus random-effects models in meta- analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97–128.
34. Sheridan, S. M., Kim, E. M., Beretvas, S. N., Smith, T., & Park, S. Y. (2017). Family–school interventions and academic and social behavioral outcomes: What matters? Paper presented at the annual meeting of American Psychological Association, Washington, DC.
35. Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30. doi:10.1002/jrsm.1091
36. Tipton, E. (2013). Robust variance estimation in meta-regression for binary dependent outcomes. Research Synthesis Methods, 4(2), 169–187. 10.1002/jrsm.1070
37. Tipton, E. (2015). Small-sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. doi:10.1037/met0000011
38. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analyses of dependent effect sizes. Behavior Research Methods, 45, 576–594. doi:10.3758/s13428-012-0261-6
39. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2014). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294.
40. Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63, 765–79.
41. Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85, 475–511.
42. Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261–293.