Statistical control versus classification of study quality in meta-analysis

Effective Education

Volumn 4, Issue 1, 2012, Pages 43-72

  Abrami, Philip C ^a   Bernard, Robert M ^a  

^a Systematic Reviews Project Manager Centre for the Study of Learning and Performance   (Canada)

Author keywords

Meta analysis; Quantitative review; Quantitative synthesis; Systematic review

Indexed keywords

EID: 84874015706     PISSN: 19415532     EISSN: 19415540     Source Type: Journal    
DOI: 10.1080/19415532.2012.761889     Document Type: Article

References (64)

1. Adesope, O.O., Lavin, T., Thompson, T., & Ungerleider, C. (2010). A systematic review and meta-analysis of the cognitive correlates of bilingualism. Review of Educational Research, 80(2), 207–245. doi: 10.3102/0034654310368803
2. Altman, D.G., Schultz, K.F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Lang, T. (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine, 134(8), 663–694.
3. Bangert-Drowns, R., Wells-Parker, E., & Chevillard, I. (1997). Assessing the methodological quality of research in narrative reviews and meta-analyses. In K.J. Bryant, M. Windle, & S.G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (pp. 405–429). Washington, DC: American Psychological Association.
4. Bernard, R.M., Abrami, P.C., Borokhovski, E., Wade, A., Tamim, R., Surkes, M., & Bethel, E.C. (2009). A meta-analysis of three interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. doi: 10.3102/0034654309333844
5. Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379–439. doi: 10.3102/00346543074003379
6. Borenstein, M., Hedges, L.V., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. Chippenham: Wiley.
7. Borenstein, M., Hedges, L.V., Higgins, J., & Rothstein, H. (2010). Comprehensive meta-analysis (Version 2.20). Englewood, NJ: Biostat.
8. Bowers, P.N., Kirby, J.R., & Deacon, S.H. (2010). The effects of morphological instruction on literacy skills: A systematic review of the literature. Review of Educational Research, 80(2), 144–179. doi: 10.3102/0034654309359353
9. Bowman, N.A. (2010). College diversity experiences and cognitive development: A meta-analysis. Review of Educational Research, 4(1), 4–33. doi: 10.3102/0034654309352495
10. Bowman, N.A. (2011). Promoting participation in a diverse democracy: A meta-analysis of college diversity experiences and civic engagement. Review of Educational Research, 81(1), 29–68. doi: 10.3102/0034654310383047
11. Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs. Chicago, IL: Rand McNally.
12. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
13. Cook, T.D. (2004). Randomized experiments in education: Why are they so rare? (Institute for Policy Research Working Paper, WP-02-19). Northwestern University, Evanston, IL.
14. Cook, T., & Campbell, D.T. (1979). Quasi-experimental design. Chicago, IL: Rand McNally.
15. Cooper, H., Allen, A.B., Patall, E.A., & Dent, A.L. (2010). Effects of full-day kindergarten on academic achievement and social development. Review of Educational Research, 80(1), 34–70. doi: 10.3102/0034654309359185
16. Credé, M., Roch, S.G., & Kieszczynka, U.M. (2010). Class attendance in college: A meta-analytic review of the relationship of class attendance with grades and student characteristics. Review of Educational Research, 80(2), 272–295. doi: 10.3102/0034654310362998
17. Denson, N. (2009). Do curricular and co-curricular diversity activities influence racial bias? A meta-analysis. Review of Educational Research, 79(2), 805–838. doi: 10.3102/0034654309331551
18. Edmonds, M.S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Tackett, K.K., & Schnakenberg, J.W. (2009). A synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers. Review of Educational Research, 79(1), 262–300. doi: 10.3102/0034654308325998
19. Eysenck, H.J. (1978). Comment: An exercise in mega-silliness. American Psychologist, 33, 517, Retrieved from http://psycnet.apa.org/journals/amp/33/5/517a.pdf
20. Field, A.P. (2003). The problems of using fixed- models of meta-analysis effects on real-world data. Understanding Statistics, 2, 77–96. doi: 10.1207/S15328031US0202_02
21. Glass, G.V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.
22. Glass, G.V. (1978). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.
23. Glazerman, S., Levy, D., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Sciences, 589, 63–93.
24. Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York, NY: Academic Press.
25. Hedges, L.V., & Pigott, T.D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203–217.
26. Hedges, L.V., & Pigott, T.D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 424–445.
27. Higgins, J.P.T., & Thompson, S.G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558.
28. Hunter, J.E., & Schmidt, F.L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Newbury Park, CA: Sage Publications.
29. Kieffer, M.J., Lesaux, N.L., Rivera, M., & Francis, D.J. (2009). Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity. Review of Educational Research, 79(3), 1168–1201. doi: 10.3102/0034654309332490
30. Lipsey, M.W., & Wilson, D.B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48(12), 1181–1209. doi:10.1037/0003-066X.48.12.1181
31. MacLehose, R.R., Reeves, B.C., Harvey, I.M., Sheldon, T.A., Russell, I.T., & Black, A.M.S. (2000). A systematic review of comparisons of effect sizes derived from randomized and non-randomised studies. Health Technology Assessment, 4(34), 1–154.
32. Marsh, H.W., Bornmann, L., Mutz, R., Daniel, H.-D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290–1326. doi: 10.3102/0034654309334143
33. Marulis, L.M., & Neuman, S.B. (2010). The effects of vocabulary intervention on young children’s word learning: A meta-analysis. Review of Educational Research, 80(3), 300–335. doi: 10.3102/0034654310377087
34. Matt, G.E., & Cook, T.D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L.V. Hedges, & J.C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 537–560). New York, NY: Russell Sage.
35. Moher, D., Jadad, A., & Tugwell, P. (1996). Assessing the quality of randomized controlled trials, Current issues and future directions. International Journal of Technology Assessment in Health Care, 12, 195–208.
36. Möller, J., Pohlmann, B., Köller, O., & Marsh, H.W. (2009). A meta-analytic path analysis of the internal/external frame of reference model of academic achievement and academic self-concept. Review of Educational Research, 79(3), 1129–1167. doi: 10.3102/0034654309337522
37. Mosteller, F., & Boruch, R. (2002). Evidence matters: Randomized trials in education research. Washington, DC: Brookings Institute Press.
38. Nave, B., Miech, E.J., & Mosteller, F. (1999). A rare design: The role of field trials in evaluating school practices. Paper presented at The American Academy of Arts and Sciences and Harvard University, Cambridge, MA.
39. Pedhazur, E.J. (1997). Multiple regression in behavioral research (3rd ed.). Orlando, FL: Harcourt Brace.
40. Pigott, T. (2006, April). Using retrospective power analysis in meta-analysis. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
41. Pigott, T. (2012). Advances in meta-analysis. New York, NY: Springer.
42. Rakes, C.R., Valentine, J.C., McGatha, M.B., & Ronau, R.N. (2010). Methods of instructional improvement in algebra: A systematic review and meta-analysis. Review of Educational Research, 80(3), 372–400. doi: 10.3102/0034654310374880
43. Raudenbush, S.W. (2005). Learning from attempts to improve schooling: The contribution of methodological diversity. Educational Researcher, 34(5), 25–31.
44. Ritter, G.W., Barnett, J.H., Denny, G.S., & Albin, G.R. (2009). The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis. Review of Educational Research, 79(1), 3–38. doi: 10.3102/0034654308325690
45. Robinson, D. (2004). An interview with Gene Glass. Educational Researcher, 33(3), 26–30.
46. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
47. Shadish, W.R., Luellen, J.K., & Clark, M.H. (2006). Propensity scores and quasi-experiments: A testimony to the practical side of Lee Sechrest. In R.R. Bootzin & P.E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 143–157). Washington, DC: American Psychological Association.
48. Shadish, W., & Myers, D. (2004, November). Campbell collaboration research design policy brief. Retrieved April 27, 2009, from http://camp.ostfold.net/resources/methods_policy_ briefs/
49. Slavin, R.E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15(9), 5–11.
50. Slavin, R.E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21.
51. Slavin, R.E. (2004). Educational research can and must address “what works” questions. Educational Researcher, 31(1), 24–26.
52. Slavin, R.E. (2008). Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5–14.
53. Slavin, R.E., Lake, C., Chambers, B., Cheung, A., & Davis, S. (2009). Effective reading programs for the elementary grades: A best-evidence synthesis. Review of Educational Research, 79(4), 1391–1466. doi: 10.3102/0034654309341374
54. Slavin, R.E., Lake, C., & Groff, C. (2009). Effective programs in middle and high school mathematics: A best-evidence synthesis. Review of Educational Research, 79(2), 839–911. doi: 10.3102/0034654308330968
55. Smith, G.C.S., & Pell, J.P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: Systematic review of ramdomised controlled trials. British Medical Journal, 327, 20–27.
56. Sosa, G.W., Berger, D.E., Saw, A.T., & Mary, J.C. (2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81(1), 97–128. doi: 10.3102/0034654310378174
57. van Steensel, R., McElvany, N., Kurvers, J., & Herppich, S. (2011). How effective are family literacy programs? Results of a meta-analysis. Review of Educational Research, 81(1), 69–96. doi: 10.3102/0034654310388819
58. Swanson, H.L., & Hsieh, C.-J. (2009). Reading disabilities in adults: A selective meta-analysis of the literature. Review of Educational Research, 79(4), 1362–1390. doi: 10.3102/0034654309350931
59. Tamim, R.M., Bernard, R.M., Borokhovski, E., Abrami, P.C., & Schmid, R.S. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4–28. doi: 10.3102/0034654310393361
60. Valentine, J.C., & Cooper, H.M. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130–149.
61. Weiser, B., & Mathes, P. (2011). Using encoding instruction to improve the reading and spelling performances of elementary students at risk for literacy difficulties: A best-evidence synthesis. Review of Educational Research, 81(2), 170–200. doi: 10.3102/0034654310396719
62. West, S., King, V., Caret, T., Lohr, K., McKoy, N., Sutton, S., & Lux, L. (2002). Systems to rate the strength of scientific evidence. Evidence report/technology assessment No. 47 AHRQ (Publication No. 02-E016). Rockville, MD: Agency of Healthcare Research and Quality.
63. What Works Clearinghouse. (2011, September). WWC procedures and standards handbook (Version 2.1). Retrieved May 30, 2012, from http://ies.ed.gov/ncee/wwc/pdf/reference: resources/wwc_procedures_v2_1_standards_handbook.pdf
64. Wortman, P.M. (1994). Judging research quality. In H. Cooper & L.V. Hedges (Eds.), The handbook of research synthesis (pp. 97–109). New York, NY: Russell Sage Foundation.