Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach

Psychological Methods

Volumn 19, Issue 2, 2014, Pages 211-229

  Cheung, Mike W L ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

Maximum likelihood; Meta analysis; R statistical environment; Structural equation modeling; Three level model

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); METHODOLOGY; MULTIVARIATE ANALYSIS; PSYCHOLOGICAL MODEL; STATISTICAL MODEL;

HUMANS; META-ANALYSIS AS TOPIC; MODELS, PSYCHOLOGICAL; MODELS, STATISTICAL; MULTIVARIATE ANALYSIS; RESEARCH DESIGN;

EID: 84902513191     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0032968     Document Type: Article

References (89)

1. Ahn, S., Ames, A. J.,&Myers, N. D. (2012). A review of meta-analyses in education: Methodological strengths and weaknesses. Review of Educational Research, 82, 436-476. doi:10.3102/0034654312458162
2. Aloe, A. M., Becker, B. J.,&Pigott, T. D. (2010). An alternative to R2 for assessing linear models of effect size. Research Synthesis Methods, 1, 272-283. doi:10.1002/jrsm.23
3. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
4. Au, K. Y. (1999). Intra-cultural variation: Evidence and implications for international business. Journal of International Business Studies, 30, 799-812. doi:10.1057/palgrave.jibs.8490840
5. Au, K.,&Cheung, M. W.-L. (2004). Intra-cultural variation and job autonomy in 42 countries. Organization Studies, 25, 1339-1362. doi: 10.1177/0170840604046345
6. Becker, B. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational Statistics, 17, 341-362. doi: 10.2307/1165128
7. Becker, B. J., Hedges, L. V.,&Pigott, T. D. (2004). Campbell Collaboration statistical analysis policy brief. Retrieved from http://www.campbellcollaboration.org/artman2/uploads/1/C2_Statistical_Analysis_Policy_Brief-2.pdf
8. Boker, S., Neale, M., Maes, H., Wilde, M., Spiegel, M., Brick, T., . . . Fox, J. (2011). OpenMx: An open source extended structural equation modeling framework. Psychometrika, 76, 306-317. doi:10.1007/s11336-010-9200-6
9. Borenstein, M., Hedges, L. V., Higgins, J. P. T.,&Rothstein, H. R. (2009). Introduction to meta-analysis. doi:10.1002/9780470743386
10. Borenstein, M., Hedges, L. V., Higgins, J. P. T.,&Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1, 97-111. doi:10.1002/jrsm.12
11. Bornmann, L., Mutz, R.,&Daniel, H.-D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226-238. doi:10.1016/j.joi.2007.03.001
12. Casella, G.,&Berger, R. L. (2002). Statistical inference (2nd ed.). Pacific Grove, CA: Brooks/Cole.
13. Cheung, M. W.-L. (2008). A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. Psychological Methods, 13, 182-202. doi:10.1037/a0013163
14. Cheung, M. W.-L. (2009). Constructing approximate confidence intervals for parameters with structural equation models. Structural Equation Modeling, 16, 267-294. doi:10.1080/10705510902751291
15. Cheung, M. W.-L. (2010). Fixed-effects meta-analyses as multiple-group structural equation models. Structural Equation Modeling, 17, 481-509. doi:10.1080/10705511.2010.489367
16. Cheung, M. W.-L. (2012). metaSEM: Meta-analysis using structural equation modeling. Retrieved from http://courses.nus.edu.sg/course/psycwlm/Internet/metaSEM/
17. Cheung, M. W.-L. (2013). Implementing restricted maximum likelihood estimation in structural equation models. Structural Equation Modeling, 20, 157-167. doi:10.1080/10705511.2013.742404
18. Cheung, M. W.-L. (in press). Multivariate meta-analysis as structural equation models. Structural Equation Modeling.
19. Cheung, M. W.-L.,&Chan, W. (2005). Meta-analytic structural equation modeling: A two-stage approach. Psychological Methods, 10, 40-64. doi:10.1037/1082-989X.10.1.40
20. Cheung, M. W.-L.,&Chan, W. (2009). A two-stage approach to synthesizing covariance matrices in meta-analytic structural equation modeling. Structural Equation Modeling, 16, 28-53. doi:10.1080/10705510802561295
21. Cheung, S. F.,&Chan, D. K. S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89, 780-791. doi:10.1037/0021-9010.89.5.780
22. Cochran, W. (1954). The combination of estimates from different experiments. Biometrics, 10, 101-129. doi:10.2307/3001666
23. Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-bystep approach (4th ed.). Los Angeles, CA: Sage.
24. Cooper, H.,&Hedges, L. V. (2009). Potentials and limitations. In H. Cooper, L. V. Hedges,&J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 561-572). New York, NY: Russell Sage Foundation.
25. Cooper, H., Valentine, J. C., Charlton, K.,&Melson, A. (2003). The effects of modified school calendars on student achievement and on school and community attitudes. Review of Educational Research, 73, 1-52. doi:10.3102/00346543073001001
26. Curran, P. J. (2009). The seemingly quixotic pursuit of a cumulative psychological science: Introduction to the special issue. Psychological Methods, 14, 77-80. doi:10.1037/a0015972
27. Enders, C. K. (2010). Applied missing data analysis. New York, NY: Guilford Press.
28. Fischer, R.,&Boer, D. (2011). What is more important for national well-being: Money or autonomy? A meta-analysis of well-being, burnout, and anxiety across 63 societies. Journal of Personality and Social Psychology, 101, 164-184. doi:10.1037/a0023663
29. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8. doi:10.3102/0013189X005010003
30. Gleser, L. J.,&Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper&L. V. Hedges (Eds.), The handbook of research synthesis (pp. 339-355). New York, NY: Russell Sage Foundation.
31. Goldstein, H. (2011). Multilevel statistical models (4th ed.). Hoboken, NJ: Wiley.
32. Graham, J. W., Olchowski, A. E.,&Gilreath, T. D. (2007). How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prevention Science, 8, 206-213. doi:10.1007/s11121-007-0070-9
33. Hanke, K.,&Fischer, R. (2012). Socioeconomical and sociopolitical correlates of interpersonal forgiveness: A three-level meta-analysis of the Enright Forgiveness Inventory across 13 societies. International Journal of Psychology. Advance online publication. doi:10.1080/00207594.2011.651086
34. Hardy, R. J.,&Thompson, S. G. (1996). A likelihood approach to metaanalysis with random effects. Statistics in Medicine, 15, 619-629. doi:10.1002/(SICI)1097-0258(19960330)15:6<619::AID-SIM188>3.0.CO;2-A
35. Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72, 320-338. doi:10.1080/01621459.1977.10480998
36. Hedges, L. V. (2009). Statistical considerations. In H. Cooper, L. V. Hedges,&J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 37-46). New York, NY: Russell Sage Foundation.
37. Hedges, L. V.,&Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6, 203-217. doi:10.1037/1082-989X.6.3.203
38. Hedges, L. V.,&Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426-445. doi: 10.1037/1082-989X.9.4.426
39. Hedges, L. V., Tipton, E.,&Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65. doi:10.1002/jrsm.5
40. Hedges, L. V.,&Vevea, J. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504. doi:10.1037/1082-989X.3.4.486
41. Higgins, J. P. T.,&Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558. doi:10.1002/sim.1186
42. Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
43. Hunter, J. E.,&Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.
44. Jackson, D., Riley, R.,&White, I. R. (2011). Multivariate meta-analysis: Potential and promise. Statistics in Medicine, 30, 2481-2498. doi: 10.1002/sim.4172
45. Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2, 61-76. doi:10.1002/jrsm.35
46. López-López, J. A., Marín-Martínez, F., Sánchez-Meca, J., Van den Noortgate, W.,&Viechtbauer, W. (2013). Estimation of the predictive power of the model in mixed-effects meta-regression: A simulation study. British Journal of Mathematical and Statistical Psychology. Advance online publication. doi:10.1111/bmsp.12002
47. MacKinnon, D. (2008). Introduction to statistical mediation analysis. New York, NY: Erlbaum.
48. Marín-Martínez, F.,&Sánchez-Meca, J. (1999). Averaging dependent effect sizes in meta-analysis: A cautionary note about procedures. The Spanish Journal of Psychology, 2, 32-38.
49. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D.,&O'Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79, 1290-1326. doi: 10.3102/0034654309334143
50. Matt, G. E.,&Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V. Hedges,&J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 537-560). New York, NY: Russell Sage Foundation.
51. Mehta, P. D.,&Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological Methods, 10, 259-284. doi:10.1037/1082-989X.10.3.259
52. Muthén, B. O.,&Muthén, L. K. (2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén&Muthén.
53. Nam, I.-S., Mengersen, K.,&Garthwaite, P. (2003). Multivariate metaanalysis. Statistics in Medicine, 22, 2309-2333. doi:10.1002/sim.1410
54. Neale, M. C.,&Miller, M. B. (1997). The use of likelihood-based confidence intervals in genetic models. Behavior Genetics, 27, 113-120. doi:10.1023/A:1025681223921
55. Nguyen, A.-M. D.,&Benet-Martínez, V. (2013). Biculturalism and adjustment: A meta-analysis. Journal of Cross-Cultural Psychology, 44, 122-159. doi:10.1177/0022022111435097
56. Owens, C. M.,&Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44, 795-805. doi:10.3758/s13428-011-0180-y
57. Patterson, H. D.,&Thompson, R. (1971). Recovery of inter-block information when block sizes are unequal. Biometrika, 58, 545-554. doi: 10.1093/biomet/58.3.545
58. Pigott, T. D. (2012). Advances in meta-analysis. doi:10.1007/978-1-4614-2278-5
59. Pinheiro, J. C.,&Bates, D. M. (2000). Mixed effects models in S and S-Plus. doi:10.1007/978-1-4419-0318-1
60. Preacher, K. J. (2011). Multilevel SEM strategies for evaluating mediation in three-level data. Multivariate Behavioral Research, 46, 691-731. doi:10.1080/00273171.2011.589280
61. Raudenbush, S. W. (2009). Analyzing effect sizes: Random effects models. In H. M. Cooper, L. V. Hedges,&J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 295-315). New York, NY: Russell Sage Foundation.
62. Raudenbush, S. W., Becker, B. J.,&Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111-120. doi:10.1037/0033-2909.103.1.111
63. Raudenbush, S. W.,&Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
64. R Development Core Team. (2013). R: A language and environment for statistical computing. Retrieved from http://www.R-project.org/
65. Riley, R. D., Lambert, P. C.,&Abo-Zaid, G. (2010). Meta-analysis of individual participant data: Rationale, conduct, and reporting. British Medical Journal, 340, c221. doi:10.1136/bmj.c221
66. Rindskopf, D. (1984). Using phantom and imaginary latent-variables to parameterize constraints in linear structural models. Psychometrika, 49, 37-47. doi:10.1007/BF02294204
67. Rosenthal, R.,&DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59-82. doi:10.1146/annurev.psych.52.1.59
68. Rosenthal, R.,&Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406. doi:10.1037/0033-2909.99.3.400
69. Rothstein, H. R., Sutton, A. J.,&Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. doi:10.1002/0470870168
70. SAS Institute. (2003). SAS/STAT software (Version 9.1) [Computer software]. Cary, NC.
71. Schafer, J. L.,&Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177. doi:10.1037/1082-989X.7.2.147
72. Schafer, J. L.,&Olsen, M. K. (1998). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33, 545-571. doi:10.1207/s15327906mbr3304_5
73. Searle, S. R., Casella, G.,&McCulloch, C. E. (1992). Variance components. doi:10.1002/9780470316856
74. Self, S. G.,&Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82, 605-610. doi:10.1080/01621459.1987.10478472
75. Shin, I.-S. (2009). Same author and same data dependence in metaanalysis. Unpublished doctoral dissertation, Florida State University, Tallahassee.
76. Skrondal, A.,&Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. doi: 10.1201/9780203489437
77. Snijders, T. A. B.,&Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Los Angeles, CA: Sage.
78. Steiger, J. H.,&Fouladi, R. T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 221-257). Mahwah, NJ: Erlbaum.
79. Stevens, J. R.,&Taylor, A. M. (2009). Hierarchical dependence in metaanalysis. Journal of Educational and Behavioral Statistics, 34, 46-73. doi:10.3102/1076998607309080
80. Stoel, R. D., Garre, F. G., Dolan, C.,&van den Wittenboer, G. (2006). On the likelihood ratio test in structural equation modeling when parameters are subject to boundary constraints. Psychological Methods, 11, 439-455. doi:10.1037/1082-989X.11.4.439
81. Takkouche, B., Cadarso-Suárez, C.,&Spiegelman, D. (1999). Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. American Journal of Epidemiology, 150, 206-215. doi:10.1093/oxfordjournals.aje.a009981
82. van Buuren, S.,&Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1-67.
83. Van den Bussche, E., Van den Noortgate, W.,&Reynvoet, B. (2009). Mechanisms of masked priming: A meta-analysis. Psychological Bulletin, 135, 452-477. doi:10.1037/a0015329
84. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2012). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods. Advance online publication. doi: 10.3758/s13428-012-0261-6
85. Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261-293. doi:10.3102/10769986030003261
86. Wilkinson, L.,&Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. doi:10.1037/0003-066X.54.8.594
87. Xiong, C., Miller, J. P.,&Morris, J. C. (2010). Measuring study-specific heterogeneity in meta-analysis: Application to an antecedent biomarker study of Alzheimer's disease. Statistics in Biopharmaceutical Research, 2, 300-309. doi:10.1198/sbr.2009.0067
88. Yuan, K.-H.,&Bentler, P. M. (2007). Robust procedures in structural equation modeling. In S.-Y. Lee (Ed.), Handbook of latent variable and related models (pp. 367-397). Amsterdam, the Netherlands: Elsevier/North-Holland.
89. Yuan, K.-H., Yang-Wallentin, F.,&Bentler, P. M. (2012). ML versus MI for missing data with violation of distribution conditions. Sociological Methods&Research, 41, 598-629. doi:10.1177/0049124112460373