Overviews in Education Research: A Systematic Review and Analysis

Review of Educational Research

Volumn 87, Issue 1, 2017, Pages 172-203

  Polanin, Joshua R ^a   Maynard, Brandy R ^b   Dell, Nathaniel A ^b  

^a Development Services Group   (United States)

^b SAINT LOUIS UNIVERSITY   (United States)

Author keywords

education research; meta analysis; overview; review of reviews

Indexed keywords

EID: 85014667311     PISSN: 00346543     EISSN: 19351046     Source Type: Journal    
DOI: 10.3102/0034654316631117     Document Type: Article

References (65)

1. American Psychological Association. (2010). Meta-Analysis Reporting Standards. Retrieved from https://www.apa.org/pubs/journals/features/arc-mars-questionnaire.pdf
2. *Anderson R. D., (1983). A consolidation and appraisal of science meta-analyses. Journal of Research in Science Teaching, 20, 497–509. doi:10.1002/tea.3660200511
3. Apple, Inc. (2014). Filemaker Pro [computer software]. Santa Clara, CA: Author.
4. Bastian H., Glasziou P., Chalmers I., (2010). Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Medicine, 7(9), e1000326. doi:10.1371/journal.pmed.1000326
5. Becker L. A., Oxman A. D., (2008). Overviews of reviews. In Higgins J. P. T., Green S., (Eds.), Cochrane handbook for systematic reviews of interventions: Cochrane book series. Chichester, England: John Wiley.
6. *Browne G., Gafni A., Roberts J., Byrne C., Majumdar B., (2004). Effective/efficient mental health programs for school-age children: A synthesis of reviews. Social Science & Medicine, 58, 1367–1384. doi:10.1016/S0277-9536(03)00332-0
7. Campbell Collaboration. (2014). Campbell collaboration systematic reviews: Policies and guidelines. Campbell Systematic Reviews: Supplement 1. doi:10.4073/csrs.2014.1
8. Chandler J., Churchill R., Higgins J., Lasserson T., Tovey D., (2013). Methodological standards for the conduct of new Cochrane Intervention Reviews. Retrieved from http://editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.2%2017122012_0.pdf
9. *Cobb B., Lehmann J., Newman-Gonchar R., Alwell M., (2009). Self-determination for students with disabilities: A narrative metasynthesis. Career Development for Exceptional Individuals, 32, 108–114. doi:10.1177/0885728809336654
10. Combs J. G., Ketchen D. J., Jr. Crook T. R., Roth P. L., (2011). Assessing cumulative evidence within “macro” research: Why meta-analysis should be preferred over vote counting. Journal of Management Studies, 48, 178–197. doi:10.1111/j.1467-6486.2009.00899.x
11. *Cook C. R., Gresham F. M., Kern L., Barreras R. B., Thornton S., Crews S. D., (2008). Social skills training for secondary students with emotional and/or behavioral disorders: A review and analysis of the meta-analytic literature. Journal of Emotional and Behavioral Disorders, 16, 131–144. doi:10.1177/1063426608314541
12. Cooper H., (2010). Research synthesis and meta-analysis (4th ed.). Thousand Oaks, CA: Sage.
13. Cooper H., Hedges L. V., Valentine J. C., (Eds.). (2009). The handbook of research synthesis and meta-analysis. Russell Sage Foundation.
14. Cooper H., Koenka A. C., (2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67, 446–462. doi:10.1037/a0027119
15. DerSimonian R., Laird N., (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188. doi:10.1016/0197-2456(86)90046-2
16. *Diekstra R. F., (2008). Effectiveness of school-based social and emotional education programmes worldwide. In Clouder C., Dahlin B., Diekstra R. F., (Eds.), Social and emotional education: An international analysis (pp. 255–312). Santender, Spain: Fundación Marcelino Botin.
17. *Forness S. R., (2001). Special education and related services: What have we learned from meta-analysis? Exceptionality, 9, 185–197. doi:10.1207/S15327035EX0904_3
18. *Fraser B. J., Walberg H. J., Welch W. W., Hattie J. A., (1987a). Contextual and transactional influences on science outcomes. International Journal of Education Research, 11, 165–186.
19. *Fraser B. J., Walberg H. J., Welch W. W., Hattie J. A., (1987b). Identifying the salient facets of a model of student learning: A synthesis of meta analyses. International Journal of Educational Research, 11, 187–212.
20. *Fraser B. J., Walberg H. J., Welch W. W., Hattie J. A., (1987c). Syntheses of research on factors influencing learning. International Journal of Education Research, 11, 155–164.
21. Gough D., Oliver S., Thomas J., (2012). An introduction to systematic reviews (1st ed.). London, England: Sage.
22. *Green J., Howes F., Waters E., Maher E., Oberklaid F., (2005). Promoting the social and emotional health of primary school-aged children: Reviewing the evidence base for school-based interventions. International Journal of Mental Health Promotion, 7, 30–36. doi:10.1080/14623730.2005.9721872
23. *Gresham F. M., (1998). Social skills training: Should we raze, remodel, or rebuild? Behavioral Disorders, 24, 19–25.
24. *Guthrie J. T., Seifert M., Mosberg L., (1983). Research synthesis in reading: Topics, audiences, and citation rates. Reading Research Quarterly, 19, 16–27. doi:10.2307/747334
25. Guyatt G. H., Oxman A. D., Vist G. E., Kunz R., Falck-Ytter Y., Alonso-Coello P., Schünemann H. J., (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336, 924 - 926. doi: 10.1136/bmj.39489.470347.AD
26. Hartling L., Chisholm A., Thomson D., Dryden D. M., (2012). A descriptive analysis of overviews of reviews published between 2000 and 2011. PLoS One, 7(11), e49667. doi:10.1371/journal.pone.0049667
27. *Hattie J., (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1), 5–13.
28. *Hattie J., (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Oxon, England: Routledge.
29. Higgins J. P., Green S., (Eds.). (2008). Cochrane handbook for systematic reviews of interventions: Cochrane book series. Chichester, England: John Wiley.
30. *Higgins S., Xiao Z., Katsipataki M., (2012). The impact of digital technology on learning: A summary for the education endowment foundation. Retrieved from http://educationendowmentfoundation.org.uk/uploads/pdf/The_Impact_of_Digital_Technology_on_Learning_-_Executive_Summary_%282012%29.pdf
31. Ioannidis J. P. A., (2009). Integration of evidence from multiple meta-analyses: A primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. Canadian Medical Association Journal, 181, 488–493. doi:10.1503/cmaj.081086
32. Kazrin A., Durac J., Agteros T., (1979). Meta-meta analysis: A new method for evaluating therapy outcome. Behaviour Research and Therapy, 17, 397–399. doi:10.1016/0005-7967(79)90011-1
33. *Kulik J. A., Kulik C. L. C., (1989). The concept of meta-analysis. International Journal of Educational Research, 13, 227–340. doi:10.1016/0883-0355(89)90052-9
34. Li L., Tian J., Tian H., Sun R., Liu Y., Yang K., (2012). Quality and transparency of overviews of systematic reviews. Journal of Evidence-Based Medicine, 5(3), 166-173.
35. *Lipsey M. W., Wilson D. B., (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181–1209. doi:10.1037//0003-066X.48.12.1181
36. Lipsey M. W., Wilson D. B., (2001). Practical meta-analysis (2nd ed.). Thousand Oaks, CA: Sage.
37. *Lister-Sharp D., Chapman S., Stewart-Brown S., Sowden A., (1999). Health promoting schools and health promotion in schools: Two systematic reviews. Health Technology Assessment, 3, 1–207.
38. *Lloyd J. W., Forness S. R., Kavale K. A., (1998). Some methods are more effective than others. Intervention in School and Clinic, 33, 195–200. doi:10.1177/105345129803300401
39. *Maag J. W., (2006). Social skills training for students with emotional and behavioral disorders: A review of reviews. Behavioral Disorders, 32, 4–17.
40. Merrell K. W., Gueldner B. A., Ross S. W., Isava D. M., (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly, 23, 26–42. doi:10.1037/1045-3830.23.1.26
41. Moher D., Liberati A., Tetzlaff J., Altman D. G., (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med, 6(6), e1000097. doi:10.1371/journal.pmed1000097
42. Pieper D., Antoine S. L., Morfeld J. C., Mathes T., Eikermann M., (2014a). Methodological approaches in conducting overviews: Current state in HTA agencies. Research Synthesis Methods, 5, 187–199. doi:10.1002/jrsm.1107
43. Pieper D., Antoine S. L., Neugebauer E. A., Eikermann M., (2014b). Systematic review finds overlapping reviews were not mentioned in every other review. Journal of Clinical Epidemiology, 67, 368–375. doi:10.1016/j.jclinepi.2013.11.007
44. Pieper D., Antoine S. L., Neugebauer E. A., Eikermann M., (2014c). Up-to-dateness of reviews is often neglected in overviews: A systematic review. Journal of Clinical Epidemiology, 67, 1302–1308. doi:10.1016/j.jclinepi.2014.08.008
45. Pieper D., Buechter R., Jerinic P., Eikermann M., (2012). Overviews of reviews often have limited rigor: A systematic review. Journal of Clinical Epidemiology, 65, 1267–1273. doi:10.1016/j.jclinepi.2012.06.015
46. Pigott T. D., (2012). Advances in meta-analysis. New York, NY: Springer.
47. Polanin J. R., Pigott T. D., (2014). The use of meta-analytic statistical significance testing. Research Synthesis Methods, 6, 63–73. doi:10.1002/jrsm.1124
48. R Core Team. (2015). R: A language and environment for statistical computing (Version 3.2.3) [Software]. Available from: https://www.r-project.org/
49. Schmidt F. L., Oh I. S., (2013). Methods for second order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121, 204–218. doi:10.1016/j.obhdp.2013.03.002
50. Shadish W. R., Lecy J. D., (2015). The meta-analytic big bang. Research Synthesis Methods. Advanced online publication. doi:10.1002/jrsm.1132
51. Shea B. J., Grimshaw J. M., Wells G. A., Boers M., Andersson N., Hamel C., … Bouter L. M., (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10. doi:10.1186/1471-2288-7-10
52. *Sipe T. A., Curlette W. L., (1997). A meta-synthesis of factors related to educational achievement: A methodological approach to summarizing and synthesizing meta-analyses. International Journal of Educational Research, 25, 583–698.
53. Smith V., Devane D., Begley C. M., Clarke M., (2011). Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology, 11, 15. doi:10.1186/1471-2288-11-15
54. *Tamim R. M., Bernard R. M., Borokhovski E., Abrami P. C., Schmid R. F., (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4–28. doi:10.3102/0034654310393361
55. *Tennant R., Goens C., Barlow J., Day C., Stewart-Brown S., (2007). A systematic review of reviews of interventions to promote mental health and prevent mental health problems in children and young people. Journal of Public Mental Health, 6, 25–32. doi:10.1108/17465729200700005
56. Thomson D., Foisy M., Oleszczuk M., Wingert A., Chisholm A., Hartling L., (2013). Overview of reviews in child health: Evidence synthesis and the knowledge base for a specific population. Evidence-Based Child Health: A Cochrane Review Journal, 8, 3–10. doi:10.1002/ebch.1897
57. Thomson D., Russell K., Becker L., Klassen T., Hartling L., (2010). The evolution of a new publication type: Steps and challenges of producing overviews of reviews. Research Synthesis Methods, 1, 198–211. doi:10.1002/jrsm.30
58. *Torgerson C. J., (2007). The quality of systematic reviews of effectiveness in literacy learning in English: A “tertiary” review. Journal of Research in Reading, 30, 287–315. doi:10.1111/j.1467-9817.2006.00318.x
59. Ttofi M. M., Farrington D. P., (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56. doi:10.1007/s11292-010-9109-1
60. Valentine J. C., Cooper H., (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130–149. doi:10.1037/10889X.13.2.130
61. *Wang M. C., Haertel G. D., Walberg H. J., (1993). Toward a knowledge base for school learning. Review of Educational Research, 63, 249–294. doi:10.3102/00346543063003249
62. *Weare K., Nind M., (2011). Mental health promotion and problem prevention in schools: What does the evidence say? Health Promotion International, 26, 29–69. doi:10.1093/heapro/dar075
63. What Works Clearinghouse. (2015). Procedures and standards handbook (Version 3). Washington, DC: Institute of Education Sciences, Department of Education. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_standards_handbook.pdf
64. Whiting P., Savović J., Higgins J. P., Caldwell D. M., Reeves B. C., Shea B., … Churchill R., (2016). ROBIS: a new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225-234.
65. Williams R. T., (2012). Using robust standard errors to combine multiple estimates with meta-analysis (Unpublished doctoral dissertation). Loyola University, Chicago, IL.