A Review of Meta-Analyses in Education: Methodological Strengths and Weaknesses

Review of Educational Research

Volumn 82, Issue 4, 2012, Pages 436-476

  Ahn, Soyeon ^a   Ames, Allison J ^a   Myers, Nicholas D ^a  

^a UNIVERSITY OF MIAMI   (United States)

Author keywords

credibility; generalizability; meta analysis; validity

Indexed keywords

EID: 84869815605     PISSN: 00346543     EISSN: None     Source Type: Journal    
DOI: 10.3102/0034654312458162     Document Type: Review

References (102)

1. *Abrami P. C.,Bernard R. M.,Borokhovski E.,Wade A.,Surkes M. A.,Tamim R.,Zhang D.Instructional interventions affecting critical thinking skills and dispositions: A Stage 1 meta-analysis.Review of Educational Research. 2008;78:1102-1134.
2. *Adesope O. O.,Lavin T.,Thompson T.,Ungerleider C.A systematic review and meta-analysis of the cognitive correlates of bilingualism.Review of Educational Research. 2010;80:207-245.
3. Aguinis H.,Dalton D. R.,Bosco F. A.,Pierce C. A.,Dalton C. M.Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact.Journal of Management. 2011;37:5-38.
4. Ahn S.,Becker B. J.Incorporating study quality in combining studies.Journal of Educational and Behavioral Statistics. 2011;36 (5): 553-585.
5. *Allen C. S.,Chen Q.,Willson V. L.,Hughes J. N.Quality of research design moderates effects of grade retention on achievement: A meta-analytic, multilevel analysis.Educational Evaluation and Policy Analysis. 2009;31:480-499.
6. Aloe A.,Becker B. Paper presented at the annual meeting of the American Educational Research Association in New York; 2009aNY; 2009a.
7. *Aloe A.,Becker B.Teacher verbal ability and school outcomes: Where is evidence?.Educational Researcher. 2009b;38:612-614.
8. American Educational Research Association. (2012a). Journals. Retrieved from http://www.aera.net/Publications/Journals/tabid/10232/Default.aspx.
9. American Educational Research Association. (2012b). Publications. Retrieved from http://www.aera.net/Publications/tabid/10067/Default.aspx.
10. Reporting standards for research in psychology: Why do we need them? What might they be?.American Psychologist. 2008;63:839-851.
11. Aytug Z. G.,Rothstein H. R.,Zhou W.,Kern M. C.Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses.Organizational Research Methods. 2012;15:103-133.
12. *Bakermans-Kranenburg M. J.,van IJzendoorn M. H.,Bradley R. H.Those who have, receive: The Matthew Effect in early childhood intervention in the home environment.Review of Educational Research. 2005;75:1-26.
13. *Bangert-Drowns R. L.,Hurley M. M.,Wilkinson B.The effects of school-based writing-to-learn interventions on academic achievement: A meta-analysis.Review of Educational Research. 2004;74:29-58.
14. Becker B. J.Intellectual talent: Psychometric and social issues. Benbow C. P.Lubinski D., ed. Baltimore, MD: Johns Hopkins Press; 1996:362-383.
15. Becker B. J.The quality and credibility of research reviews: What the editors say.Personality and Social Psychology Bulletin. 2003;17:267-272.
16. Begg C. B.Handbook of research synthesis. Cooper H.Hedges L. V., ed. New York, NY: Russell Sage Foundation; 1994:399-409.
17. *Bernard R. M.,Abrami P. C.,Borokhovski E.,Wade C. A.,Tamim R. M.,Surkes M. A.,Bethel E. C.A meta-analysis of three types of interaction treatments in distance education.Review of Educational Research. 2009;79:1243-1289.
18. *Bernard R. M.,Abrami P. C.,Lou Y.,Borokhovski E.,Wade A.,Wozney L.,Huang B.How does distance education compare with classroom instruction? A meta-analysis of the empirical literature.Review of Educational Research. 2004;74:379-439.
19. Birge R. T.The calculation of errors by the method of least squares.Reviews of Modern Physics. 1932;40:207-227.
20. Borenstein M.,Hedges L. V.,Higgins J. P. T.,Rothstein H. R.Introduction to meta-analysis. West Sussex, UK: Wiley; 2009:.
21. *Borman G. D.,Dowling N. M.Teacher attrition and retention: A meta-analytic and narrative review of the research.Review of Educational Research. 2008;78:367-409.
22. *Borman G. D.,Hewes G. M.,Overman L. T.,Brown S.Comprehensive school reform and achievement: A meta-analysis.Review of Educational Research. 2003;73:125-230.
23. *Bowers P. N.,Kirby J. R.,Deacon S. H.The effects of morphological instruction on literacy skills.Review of Educational Research. 2010;80:144-179.
24. *Bowman N. A.College diversity experiences and cognitive development: A meta-analysis.Review of Educational Research. 2010;80:4-33.
25. Bracht G. H.,Glass G. V.The external validity of experiments.American Educational Research Journal. 1968;5:437-474.
26. Briggs D. C.Synthesizing causal inferences.Educational Researcher. 2008;37:15-22.
27. Cafri G.,Kromrey J. D.,Brannick M. T.A meta-meta-analysis: Empirical review of statistical power, type I error rates, effect sizes, and model selection of meta-analyses published in psychology.Multivariate Behavioral Research. 2010;45:239-270.
28. Campbell D. T.,Stanley J.Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally; 1976:.
29. Cohen J.A coefficient of agreement for nominal scales.Educational and Psychological Measurement. 1960;20:37-46.
30. Cohen J.Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit.Psychological Bulletin. 1968;70:213-220.
31. Cook T. D.,Campbell D. T.Quasi-experimentation: Design & analysis issues for field settings. Boston, MA: Houghton Mifflin Co; 1979:.
32. Cooper H. M.Scientific guidelines for conducting integrative research reviews.Review of Educational Research. 1982;52:291-302.
33. Cooper H. M.Synthesizing research: A guide for literature reviews. 1998). Synthesizing research: A guide for literature reviews (3rd ed.). Newbury Park, CA: Sage.Newbury Park, CA: Sage; 1998:.
34. Cooper H. M.Research synthesis and meta-analysis: A guide for literature review. 2009). Research synthesis and meta-analysis: A guide for literature review (4th ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2009:.
35. *Cooper H. M.,Allen A. B.,Patall E. A.,Dent A. L.Effects of full-day kindergarten on academic achievement and social development.Review of Educational Research. 2010;80:34-70.
36. Cooper H. M.,Hedges L. V.,Valentine J. C.The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation; 2009:.
37. *Cooper H. M.,Robinson J. C.,Patall E. A.Does homework improve academic achievement? A synthesis of research, 1987-2003.Review of Educational Research. 2006;76:1-62.
38. *Cooper H. M.,Valentine J. C.,Charlton K.,Melson A.The effects of modified school calendars on student achievement and on school and community attitudes.Review of Educational Research. 2003;73:1-52.
39. *Cornelius-White J.Learner-centered teacher-student relationships are effective: A meta-analysis.Review of Educational Research. 2007;77:113-143.
40. *Credé M.,Roch S. G.,Kieszczynka U. M.Class attendance in college.Review of Educational Research. 2010;80:272-295.
41. Cronbach L. J.Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass; 1982:.
42. *D'Agostino J. V.,Murphy J. A.A meta-analysis of reading recovery in United States schools.Educational Evaluation and Policy Analysis. 2004;26:23-38.
43. *D'Agostino J. V.,Powers S. J.Predicting teacher performance with test scores and grade point average: A meta-analysis.American Educational Research Journal. 2009;46:146-182.
44. DeCoster J.The encyclopedia of social measurement. Kempf-Leonard K., ed. San Diego, CA: Academic Press; 2004:1-19.
45. *Denson N.Do curricular and cocurricular diversity activities influence racial bias? A meta-analysis.Review of Educational Research. 2009;79:805-838.
46. Dynarski S.Building the stock of college-educated labor.Journal of Human Resources. 2008;43:576-610.
47. *Edmonds M. S.,Vaughn S.,Wexler J.,Reutebuch C.,Cable A.,Tackett K. K.,Schnakeberg J.A synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers.Review of Educational Research. 2009;79:262-300.
48. *Ehri L. C.,Nunes S. R.,Stahl S. A.,Willows D. M.Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel's meta-analysis.Review of Educational Research. 2001;71:393-447.
49. *Elbaum B.,Vaughn S.,Hughes M. T.,Moody S. W.How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research.Journal of Educational Psychology. 2000;92:605-619.
50. *Falchikov N.,Goldfinch J.Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks.Review of Educational Research. 2000;70:287-322.
51. *Gijbels D.,Dochy F.,Van den Bossche P.,Segers M.Effects of problem-based learning: A meta-analysis from the angle of assessment.Review of Educational Research. 2005;75:27-61.
52. *Ginsburg-Block M. D.,Rohrbeck C. A.,Fantuzzo J. W.A meta-analytic review of social, self-concept, and behavioral outcomes for peer-assisted learning.Journal of Educational Psychology. 2006;98:732-749.
53. *Graham S.,Perin D.A meta-analysis of writing instruction for adolescent students.Journal of Educational Psychology. 2007;99:445-476.
54. Harwell M.,Maeda Y.Deficiencies of reporting in meta-analyses and some remedies.The Journal of Experimental Education. 2008;76:403-430.
55. Hedges L. V.,Pigott T.The power of statistical tests in meta-analysis.Psychological Methods. 2001;6:203-217.
56. Hedges L. V.,Pigott T.The power of statistical tests for moderators in meta-analysis.Psychological Methods. 2004;9:426-445.
57. Higgins J. P. T.,Thompson S. G.Quantifying heterogeneity in a meta-analysis.Statistics in Medicine. 2002;21:1539-1558.
58. Hunter J. E.,Schmidt F. L.Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage; 1990:.
59. Hunter J. E.,Schmidt F. L.Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage; 2004:.
60. *Jones S. M.,Dindia K.A meta-analytic perspective on sex equity in the classroom.Review of Educational Research. 2004;74:443-471.
61. *Kieffer M. J.,Lesaux N. K.,Rivera M.,Francis D. J.Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity.Review of Educational Research. 2009;79:1168-1201.
62. *Klauer K. J.,Phye G. D.Inductive reasoning: A training approach.Review of Educational Research. 2008;78:85-123.
63. Kulik J. A.,Kulik C. C.,Cohen P. A.Effectiveness of computer-based college teaching: A meta-analysis of findings.Review of Educational Research. 1980;50:525-544.
64. *Kuncel N. R.,Credé M.,Thomas L. L.The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature.Review of Educational Research. 2005;75:63-82.
65. *La Paro K. M.,Pianta R. C.Predicting children's competence in the early school years: A meta-analytic review.Review of Educational Research. 2000;70:443-484.
66. *Lauer P. A.,Akiba M.,Wilkerson S. B.,Apthorp H. S.,Snow D.,Martin-Glenn M. L.Out-of-school-time programs: A meta-analysis of effects for at-risk students.Review of Educational Research. 2006;76:275-313.
67. Lipsey M.,Wilson D.Wilderness challenge programs for delinquent youth: A meta-analysis of outcome evaluations.Evaluation and Program Planning. 2000;23:1-12.
68. *Lou Y.,Abrami P. C.,d'Apollonia S.Small group and individual learning with technology: A meta-analysis.Review of Educational Research. 2001;71:449-521.
69. *Marsh H. W.,Bornmann L.,Mutz R.,Daniel H.-D.,O'Mara A.Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches.Review of Educational Research. 2009;79:1290-1326.
70. *Marulis L. M.,Neuman S. B.The effects of vocabulary intervention on young children's word learning.Review of Educational Research. 2010;80:300-335.
71. Matt G. E.,Cook T. D.The handbook of research synthesis. Cooper H.Hedges L. V., ed. New York, NY: Russell Sage Foundation; 1994:503-520.
72. McDonald S.-K.,Keesler V. A.,Kauffman N. J.,Schneider B.Scaling-up exemplary interventions.Educational Researcher. 2006;35:15-24.
73. *Metzger S. A.,Wu M.-J.Commercial teacher selection instruments: The validity of selecting teachers through beliefs, attitudes, and values.Review of Educational Research. 2008;78:921-940.
74. Moher D.,Cook D. J.,Eastwood S.,Olkin I.,Rennie D.,Stroup D.Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement.Lancet. 1999;354:1896-1900.
75. *Mol S. E.,Bus A. G.,de Jong M. T.Interactive book reading in early education: A tool to stimulate print knowledge as well as oral language.Review of Educational Research. 2009;79:979-1007.
76. *Murphy P. K.,Wilkinson I. A. G.,Soter A. O.,Hennessey M. N.,Alexander J. F.Examining the effects of classroom discussion on students' comprehension of text: A meta-analysis.Journal of Educational Psychology. 2009;101:740-764.
77. *Nesbit J. C.,Adesope O. O.Learning with concept and knowledge maps: A meta-analysis.Review of Educational Research. 2006;76:413-448.
78. Orwin R. G.The handbook of research synthesis. Cooper H.Hedges L. V., ed. New York, NY: Russell Sage Foundation; 1994:139-162.
79. *Patall E. A.,Cooper H.,Robinson J. C.Parent involvement in homework: A research synthesis.Review of Educational Research. 2008;78:1039-1101.
80. *Penny A. R.,Coe R.Effectiveness of consultation on student ratings feedback: A meta-analysis.Review of Educational Research. 2004;74:215-253.
81. *Rakes C. R.,Valentine J. C.,McGatha M. B.,Ronau R. N.Methods of instructional improvement in algebra.Review of Educational Research. 2010;80:372-400.
82. *Ritter G. W.,Barnett J. H.,Denny G. S.,Albin G. R.The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis.Review of Educational Research. 2009;79:3-38.
83. *Rohrbeck C. A.,Ginsburg-Block M. D.,Fantuzzo J. W.,Miller T. R.Peer-assisted learning interventions with elementary school students: A meta-analytic review.Journal of Educational Psychology. 2003;95:240-257.
84. Schmidt F. L.,Le H.Software for the Hunter-Schmidt meta-analysis methods. Iowa City, IA: University of Iowa, Department of Management & Organizations; 2004:.
85. *Sénéchal M.,Young L.The effect of family literacy interventions on children's acquisition of reading from kindergarten to grade 3: A meta-analytic review.Review of Educational Research. 2008;78:880-907.
86. Shadish W. R.,Cook T. D.,Campbell D. T.Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin; 2002:.
87. *Sirin S. R.Socioeconomic status and academic achievement: A meta-analytic review of research.Review of Educational Research. 2005;75:417-453.
88. Slavin R. E.What works? Issues in synthesizing educational program evaluations.Educational Researcher. 2008;37:5-14.
89. *Slavin R. E.,Cheung A.A synthesis of research on language of reading instruction for English language learners.Review of Educational Research. 2005;75:247-284.
90. *Slavin R. E.,Lake C.Effective programs in elementary education: A best-evidence synthesis.Review of Educational Research. 2008;78:427-515.
91. *Slavin R. E.,Lake C.,Chambers B.,Cheung A.,Davis S.Effective reading programs for the elementary grades: A best-evidence synthesis.Review of Educational Research. 2009;79:1391-1466.
92. *Slavin R. E.,Lake C.,Groff C.Effective programs in middle and high school mathematics: A best-evidence synthesis.Review of Educational Research. 2009;79:839-911.
93. *Stuebing K. K.,Fletcher J. M.,LeDoux J. M.,Lyon R.,Shaywitz S. E.,Bennett A. S.Validity of IQ-discrepancy classifications of reading disabilities: A meta-analysis.American Educational Research Journal. 2002;39:469-518.
94. *Swanson H. L.,Hsieh C.-J.Reading disabilities in adults: A selective meta-analysis of the literature.Review of Educational Research. 2009;79:1362-1390.
95. *Swanson H. L.,Jerman O.Math disabilities: A selective meta-analysis of the literature.Review of Educational Research. 2006;76:249-274.
96. *Swanson H. L.,Lussier C. M.A selective synthesis of the experimental literature on dynamic assessment.Review of Educational Research. 2001;71:321-363.
97. *Swanson H. L.,Trainin G.,Necoechea D. M.,Hammill D. D.Rapid naming, phonological awareness, and reading: A meta-analysis of the correlation evidence.Review of Educational Research. 2003;73:407-440.
98. *Tenenbaum H. R.,Ruck M. D.Are teachers' expectations different for racial minority than for European American students? A meta-analysis.Journal of Educational Psychology. 2007;99:253-273.
99. Thompson B. Keynote address at the annual meeting of the Mid-Western Educational Research Association (MWERA); 2007St. Louis, MO; 2007.
100. Valentine J. C.,Cooper H.A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (study DIAD).Psychological Methods. 2008;13:330-349.
101. Valentine J. C.,Cooper H. M.,Patall E. A.,Tyson D.,Robinson J. C.A method for evaluating research syntheses: The quality, conclusions and consensus of 12 syntheses of the effects of after-school programs.Research Synthesis Methods. 2010;1:20-38.
102. Wortman P. M.The handbook of research synthesis. Cooper H.Hedges L. V., ed. New York, NY: Russell Sage Foundation; 1994:139-162.