Multilevel meta-analysis: A comparison with traditional meta-analytical procedures

Educational and Psychological Measurement

Volumn 63, Issue 5, 2003, Pages 765-790

  Van Den Noortgate, Wim ^a   Onghena, Patrick ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Maximum likelihood; Meta analysis; Monte carlo simulation; Multilevel; Random effects

Indexed keywords

EID: 0141480843     PISSN: 00131644     EISSN: None     Source Type: Journal    
DOI: 10.1177/0013164402251027     Document Type: Article

References (58)

1. Bosker, R. J., & Snijders, T. A. B. (1990). Statistische aspecten van multi-niveau onderzoek. Tijdschrift Voor Onderwijsresearch, 15, 317-329.
2. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
3. Capraro, M. M., Capraro, R. M., & Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating scale across studies. Educational and Psychological Measurement, 61, 373-386.
4. Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
5. Caruso, J. C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization (RG) study. Educational and Psychological Measurement, 61, 675-682.
6. Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101-129.
7. Cornwell, J. M., & Ladd, R. T. (1993). Power and accuracy of the Schmidt and Hunter meta-analytic procedures. Educational and Psychological Measurement, 53, 877-895.
8. Cronbach, L. J. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.
9. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177-188.
10. Draper, D. (1995). Inference and hierarchical modeling in the social sciences. Journal of Educational and Behavioral Statistics, 20(2), 115-147.
11. Erez, A., Bloom, M. C., & Wells, M. T. (1996). Using random rather than fixed effects models in meta-analysis: Implications for situational specificity and validity generalization. Personnel Psychology, 49, 275-306.
12. Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
13. Glass, G. V (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.
14. Glass, G. V, MacGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
15. Goldstein, H. (1995). Multilevel statistical models. London: Edward Arnold.
16. Goldstein, H., Rasbash, J., Plewis, I., Draper, D., Browne, W., Yang, M., et al. (1998). A user's guide to MLwiN. London: Multilevel Models Project, University of London.
17. Goldstein, H., & Yang, M. (2000). Meta-analysis using multilevel models with an application to the study of class size effects. Journal of the Royal Statistical Society, Series C: Applied Statistics, 49, 339-412.
18. Harwell, M. (1997). An empirical study of Hedges's homogeneity test. Psychological Methods, 2, 219-231.
19. Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128.
20. Hedges, L. V. (1983). A random-effects model for effect sizes. Psychological Bulletin, 93, 388-395.
21. Hedges, L. V. (1994). Fixed effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-300). New York: Russell Sage Foundation.
22. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
23. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486-504.
24. Helms, J. E. (1999). Another meta-analysis of the White Racial Identity Attitude scale's Cronbach alphas: Implications for validity. Measurement and Evaluation in Counseling and Development, 32, 122-137.
25. Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study of the Teacher Efficacy scale and related instruments. Educational and Psychological Measurement, 61, 404-420.
26. Hox, J. J., & de Leeuw, E. D. (1997). Meta-analyse via multiniveau-modellen. Tijdschrift Voor Onderwijsresearch, 22, 16-28.
27. Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
28. Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta-analysis: Cumulating findings across research. Beverly Hills, CA: Sage.
29. Johnson, B. T., Mullen, B., & Salas, E. (1995). Comparison of three major meta-analytic approaches. Journal of Applied Psychology, 80, 94-106.
30. Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. London: Sage.
31. Meijer, E., van der Leeden, R., & Busing, F. M. T. A. (1995). Implementing the bootstrap for multilevel models. Multilevel Modelling Newsletter, 7(2), 7-11.
32. Mengersen, K. L., Tweedie, R. L., & Biggerstaff, B. J. (1995). The impact of method choice in meta-analysis. Australian Journal of Statistics, 37, 19-44.
33. Mok, M. (1995). Sample size requirements for 2-level designs in educational research. Multilevel Modelling Newsletter, 7(2), 11-15.
34. Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review Public Health, 17, 1-23.
35. National Research Council. (1992). Combining information: Statistical issues and opportunities for research, Washington, DC: National Academy Press.
36. Olkin, I. (1990). History and goals. In K. W. Wachter & M. L. Straf (Eds.), The future of meta-analysis (pp. 3-10). New York: Russell Sage Foundation.
37. Overton, R. C. (1998). A comparison of fixed-effects and mixed (random effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354-379.
38. Poole, C., & Greenland, S. (1999). Random-effects meta-analyses are not always conservative. American Journal of Epidemiology, 150, 469-475.
39. Raudenbush, S. W, & Bryk, A. S. (1985). Empirical Bayes meta-analysis. Journal of Educational Statistics, 10, 75-98.
40. Rosenthal, R. (1961, August). On the social psychology of the psychological experiment: With particular reference to experimenter bias. Paper presented at the meeting of the American Psychological Association, New York.
41. Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage.
42. Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345 studies. Behavioral and Brain Sciences, 3, 377-415.
43. Rosenthal, R., & Rubin, D. B. (1982). Comparing effect sizes of independent studies. Psychological Bulletin, 92, 500-504.
44. Rubin, D. B. (1981). Estimation in parallel randomized experiments. Journal of Educational Statistics, 6, 377-400.
45. Sagie, A., & Koslowsky, M. (1993). Detecting moderators with meta-analysis: an evaluation and comparison of techniques. Personnel Psychology, 46, 629-640.
46. Sanchez-Meca, J., & Marin-Martinez, F. (1998). Testing continuous moderators in meta-analysis: A comparison of procedures. British Journal of Mathematical and Statistical Psychology, 51, 311-326.
47. Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
48. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
49. Takkouche, B., Cadarso-Suarez, C., & Spiegelman, D. (1999). Evaluation of old and new tests of heterogeneity in epidemiologic meta-analysis. American Journal of Epidemiology, 150, 206-215.
50. Thompson, S. G., & Pocock, S. J. (1991). Can meta-analyses be trusted? Lancet, 338, 1127-1130.
51. Turner, R., Omar, R., Yang, M., Goldstein, H., & Thompson, S. (2000). Multilevel models for meta-analysis of clinical trials with binary outcomes. Retrieved September 2000 from http:// www.ioe.ac.uk/multilevel/meta_analysis_for_clinical_trials.pdf
52. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
53. Vacha-Haase, T., Kogan, L., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59.
54. Van den Noortgate, W., & Onghena, P. (2001). Estimating the standardized mean difference in a meta-analysis: Bias, precision and mean squared error. Manuscript submitted for publication.
55. Viswesvaran, C., & Ones, D. (2000). Measurement error in "Big Five Factors" personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
56. Viswesvaran, C., & Sanchez, J. I. (1998). Moderator search in meta-analysis: A review and cautionary note on existing approaches. Educational and Psychological Measurement, 58, 77-87.
57. Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
58. Zeger, S., & Karim, M. (1991). Generalized linear models with random effects: A Gibbs sampling approach. Journal of the American Statistical Association, 86, 79-86.