Developments and trends in synthesizing diverse forms of evidence: Beyond comparisons between distance education and classroom instruction

Distance Education

Volumn 31, Issue 3, 2010, Pages 231-256

  Bethel, Edward C ^a   Bernard, Robert M ^a  

^a Systematic Reviews Project Manager Centre for the Study of Learning and Performance   (Canada)

Author keywords

Distance education; E learning; Online learning; Research synthesis; Systematic review

Indexed keywords

EID: 77957957622     PISSN: 01587919     EISSN: 14750198     Source Type: Journal    
DOI: 10.1080/01587919.2010.513950     Document Type: Article

References (85)

1. Abrami, P.C., & Bernard, R.M. (2006). Research on distance education: In defense of field experiments. Distance Education, 27(1), 5-26. doi: 10.1080/01587910600653116
2. Abrami, P.C., Bernard, R.M., & Wade, C.A. (2006). Affecting policy and practice: Issues involved in developing an Argument Catalogue. Evidence & Policy: A Journal of Research, Debate and Practice, 2(4), 417-437. doi: 10.1332/174426406778881737
3. Abrami, P.C., Bernard, R., Wade, A., Schmid, R.F., Borokhovski, E., Tamin, R., Surkes, M., Lowerison, G., Zhang, D., Nicolaidou, I., Newman, S., Wozney, L., & Peretiatkowicz, A. (2006). A review of e-learning in Canada: A rough sketch of the evidence, gaps and promising directions. Canadian Journal of Learning and Technology/La revue Canadienne de l'apprentissage et de la technologie, 32(3). Retrieved 1 September 2010 from http:// www.cjlt.ca/index.php/cjlt/article/view/27/25
4. Abrami, P.C., Borokhovski, E., Bernard, R.M., Wade, C.A., Tamim, R., Persson, T., Bethel, E.C., Hanz, K., & Surkes, M.A. (2010). Issues in conducting and disseminating brief reviews of evidence. Evidence and Policy: A Journal of Research, Debate and Practice, 6(3), 371-389. doi: 10.1332/174426410X524866
5. Allen, M., Bourhis, J., Mabry, E., Burrell, N., & Mattrey, E. (2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: A metaanalysis. American Journal of Distance Education, 16(2), 83-97. doi: 10.1207/ S15389286AJDE1602_3
6. Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004). Evaluating the effectiveness of distance learning: A comparison using meta-analysis. Journal of Communication, 54(3), 402-420. doi: 10.1111/j.1460-2466.2004.tb02636.x
7. Anderson, T. (2003). Getting the mix right: An updated and theoretical rationale for interaction. International Review of Research in Open and Distance learning, 4(2). Retrieved 7 September 2010 from http://www.irrodl.org/index.php/irrodl/article/view/149/230
8. Bernard, R.M., & Abrami, P.C. (2009). Meta analysis of distance education and online learning research: Past, present and future. Invited presentation at the 15th annual Sloan-C International Conference on Online Learning, 28-30 October 2009, Orlando, FL.
9. Bernard, R.M., Abrami, P.C., Borokhovski, E., Wade, A., Tamim, R., Surkes, M., & Bethel, E.C. (2009). A meta-analysis of three interaction treatments in distance education. Review of Educational Research, 79(3), 1243-1289. doi: 10.3102/0034654309333844
10. Bernard, R.M., Abrami, P.C., Lou, Y., & Borokhovski, E. (2004). A methodological morass? How we can improve the quality of quantitative research in distance education. Distance Education, 25(2), 176-198. doi: 10.1080/0158791042000262094
11. Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M. & Huang, B. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research 3(74), 379-439. doi: 10.3102/00346543074003379
12. Best Evidence Encyclopedia. (n.d.). Criteria for inclusion in the Best Evidence Encyclopedia. Baltimore, MD: Johns Hopkins University. Retrieved from http://www.bestevidence.org/methods/criteria.htm
13. Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002). Using metaethnographyto synthesize qualitative research: A worked example. Journal of Health Services Research and Policy, 7(4), 209-215. doi: 10.1258/135581902320432732
14. Brunton, G., Thomas, J., Harden, A., Rees, R., Kavanagh, J., Oliver, S., et al. (2005). Promotingphysical activity amongst children outside of physical education classes: A systematic review integrating intervention studies and qualitative studies. Health Education Journal, 64(4), 323-338. doi: 10.1177/001789690506400404
15. Campbell Collaboration. (n.d.). The Campbell Collaboration. Retrieved from http://www.campbellcollaboration.org
16. Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
17. Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
18. Cavanaugh, C.S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7(1), 73-88. Retrieved from http://www.editlib.org/j/IJET
19. Cavanaugh, C., Gillan, K.J., Kromrey, J., Hess, M., & Blomeyer, R. (2004). The effects o f distance education on K-12 student outcomes: A meta-analysis. Naperville, IL: Learning Point Associates.
20. Coalition for Evidence-Based Policy. (2008). Social programs that work. Retrieved from http://www.evidencebasedprograms.org/
21. Cochrane Collaboration. (2010). The Cochrane Collaboration. Retrieved from http://www.cochrane.org/
22. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
23. Cook, D.A., Levinson, A.J., Garside, S., Dupras, D.M., Erwin, P.J., & Montori, V. (2008). Internet-based learning in the health professions: A meta-analysis. Journal of the American Medical Association, 300(10), 1181-1196. doi: 10.1001/jama.300.10.1181
24. Cooper, H.M., Hedges, L.V., & Valentine, J.C. (2009). The handbook of research synthesis and meta-analysis. Thousand Oaks, CA: Sage.
25. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy, 10(1), 45-53.
26. Dixon-Woods, M., Agarwal, S., Young, B., Jones, D., & Sutton, A. (2004). Integrative approaches to qualitative and quantitative evidence. London: Health Development Agency.
27. Dixon-Woods, M., Shaw, R.L., Agarwal, S., & Smith, J.A. (2004). The problem of appraising qualitative research. Quality and Safety in Health Care, 13(3), 223-225. doi:10.1136/qshc.2003.008714
28. Evans, D. (2003). Hierarchy of evidence: A framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing, 12(1), 77-84. doi: 10.1046/j.1365-2702.2003.00662.x
29. Evidence for Policy and Practice Information and Co-ordinating Centre. (2006-2009). Welcome to EPPI-Centre. Retrieved from http://eppi.ioe.ac.uk/cms/
30. Eysenck, H.J. (1978). An exercise in mega-silliness. American Psychologist, 33(5), 517. doi:10.1037/0003-066X.33.5.517.a
31. Freeman, M., deMarrais, K., Preissle, J., Roulston, K., & St Pierre, E.A. (2007). Standards of evidence in qualitative research: An incitement to discourse. Educational Researcher, 36(1), 25-32. doi: 10.3102/0013189X06298009
32. Friedman, L. (2001). Why vote-count reviews don't count. Biological Psychiatry, 49(2), 161-162. Retrieved from http://www.journals.elsevierhealth.com/periodicals/bps/home
33. Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory. New York, NY: Aldine de Gruyter.
34. Glass, G.V. (1976). Primary, secondary and meta-analysis of research. EducationalResearcher, 5(10), 3-8. doi: 10.3102/0013189X005010003
35. Goldsmith, M.R., Bankhead, C.R., & Austoker, J. (2007). Synthesising quantitative and qualitative research in evidence-based patient information. British Medical Journal, 61(3), 262-270. doi: 10.1136/jech.2006.046110
36. Gough, D. (2007). Weight of evidence: A framework for the appraisal of the quality and relevance of evidence. Research Papers in Education, 22(2), 213-228. doi:10.1080/02671520701296189
37. Green, J.L., & Skukauskait, A. (2008). Comments on Slavin: Becoming critical readers: Issues in transparency, representation, and warranting of claims. Educational Researcher, 37(1), 30-40. doi:10.3102/0013189X08314828
38. Greenhalgh, T., Robert, G., MacFarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629. Retrieved from http://www.milbank.org/quarterly.html
39. Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., & Oakley, A. (2004). Applying systematic review methods to studies of people's views: An example from public health research. Journal of Epidemiology & Community Health, 58(9), 794-800. doi:10.1136/jech.2003.014829
40. Hedges, L.V., & Olkin, I. (1980). Vote-counting methods in research synthesis. Psychological Bulletin, 88(2), 359-369. doi: 10.1037/0033-2909.88.2.359
41. Higgins, J.P.T., & Green S. (Eds.). (2008). Cochrane handbook for systematic reviews of interventions version 5.0.1 [updated September 2008]. Retrieved from http://wwwcochrane-handbook.org
42. Jahng, N., Krug, D., & Zhang, Z. (2007). Student achievement in online education compared to face-to-face education. European Journal of Open, Distance and E-Learning. Retrieved from http://www.eurodl.org/materials/contrib/2007/Jahng_Krug_Zhang.htm
43. Larsson, R. (1993). Case survey methodology: Quantitative analysis of patterns across case studies. The Academy of Management Journal, 36(6), 1515-1546. Retrieved from http://www.jstor.org/stable/256820
44. Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage Publications.
45. Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance education: A theory-based meta-analysis of empirical literature. Educational Technology Research & Development, 5(2), 141-176. doi: 10.1007/s11423-006-8252-x
46. Machtmes, K., & Asher, J.W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. American Journal of Distance Education, 14(1), 27-46. doi:10.1080/08923640009527043
47. Major, C.H. (2010). Do virtual professors dream of electric students? University faculty experiences with online distance education. Teachers College Record, 112(8). Retrieved from http://www.tcrecord.org
48. Maxwell, J.A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279-300. Retrieved from http://www.hepg.org/her
49. Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research & Policy, 10(Supplement 1), 6-20. doi: 10.1258/1355819054308576
50. Noblit, G.W., & Hare, R.D. (1988). Meta-ethnography: Synthesizing qualitative studies. Thousand Oaks, CA: Sage.
51. Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review - A new method of systematic review designed for complex policy interventions. Journal of Health Services Research and Policy, 10(Supplement 1), 21-34. doi: 10.1258/1355819054308530
52. Petticrew, M., & Roberts, H. (2003). Evidence, hierarchies, and typologies: horses for courses. British Medical Journal, 57(7), 527-529. doi: 10.1136/jech.57.7.527
53. Pope, C., Mays, N., & Popay, J. (2007). Synthesizing qualitative and quantitative health research: A guide to methods. Milton Keynes: Open University Press.
54. Rees, R., Kavanagh, J., Harden, A., Shepherd, J., Brunton, G., Oliver, S., & Oakley, A. (2006). Young people and physical activity: A systematic review matching their views to effective interventions. Health Education Research, 21(6), 806-825. doi: 10.1093/her/cyl120
55. Robinson, D.H. (2004). An interview with Gene V. Glass. Educational Researcher, 33(3),26-30. doi: 10.3102/0013189X033003026
56. Russell, T.L. (2001). The no significant difference phenomenon. Retrieved from http://www.nosignificantdifference.org/
57. Sandelowski, M. (2004). Using qualitative research. Qualitative Health Research, 14(10),1366-1386. doi: 10.1177/1049732304269672
58. Sandelowski, M., Docherty, S., & Emden, C. (1997). Qualitative metasynthesis: Issues and techniques. Research in Nursing & Health, 20(4), 365-371. doi: 10.1002/(SICI)1098-240X (199708)20:4<365::AID-NUR9>3.0.CO;2-E
59. Sandelowski, M., Voils, C.I., & Barroso, J. (2006). Defining and designing mixed research synthesis studies. Research in the Schools, 13(1), 29-40. Retrieved from http://www.msera.org/rits.htm
60. Sandelowski, M., Voils, C.I., & Barroso, J. (2007). Comparability work and the management of difference in research synthesis studies. Social Science & Medicine, 64(1), 236-247. doi:10.1016/j.socscimed.2006.08.041
61. Schmid, R.F., Bernard, R.M., Borokhovski, E., Tamim, R., Abrami, P.C., Wade, C.A., Surkes, M.A., & Lowerison, G. (2009). Technology's effect on achievement in higher education: A Stage I meta-analysis of classroom applications. Journal of Computing in Higher Education, 21(2), 95-109. doi: 10.1007/s12528-009-9021-8
62. Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance educationacademic performances: A meta-analytic approach. International Review of Research in Open and Distance Learning, 4(2). Retrieved 8 September 2010 from http://www.irrodlorg/index.php/irrodl/article/view/153/234
63. Shadish, W., & Myers, D. (2004). Campbell collaboration research policy brief: Methods policy briefs. Retrieved from http://camp.ostfold.net/artman2/uploads/1/Research_Design Policy_Brief.pdf
64. Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59(3), 623-664. doi: 10.1111/j.1744-6570.2006.00049.x
65. Slavin, R.E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15(9), 5-11. doi: 10.3102/0013189X015009005
66. Slavin, R.E. (2008). Perspectives on evidence-based research in education - What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5-14. doi:10.3102/0013189X08314117
67. Smith, G.C.S. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: Systematic review of randomised controlled trials. British Medical Journal, 327, Number 7429, 1459-1461. doi: 10.1136/bmj.327.7429.1459
68. So, H.-J., & Kim, B. (2005). Instructional methods for CSCL: Review of case studies. In Proceedings of the 2005 conference on Computer support for collaborative learning: learning 2005: The next 10 years! 30 May- 4 June 2005 Retrieved from http://portal.acm.org/toc.cfm?id= 1149293
69. Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17). Retrieved from http://PAREonline.net/getvn.asp?v=7&n=17
70. Strauss, A.L., & Corbin, J.M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: Sage.
71. Strobel, J., & van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. The Interdisciplinary Journal of Problem-based Learning, 3(1), 44-58. Retrieved from http://docs.lib.purdue.edu/ijpbl/
72. Tamim, R., Bernard, R.M., Borokhovski, E., Abrami, P.C., & Schmid, R.F. (2009). Effects of technology on students' achievement: A second-order meta-analysis. Paper presented at the Campbell Collaboration Ninth Annual Colloquium, 18-20 May 2009, Oslo, Norway.
73. Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), 45. Retrieved from http://www.biomedcentral.com/1471-2288/8/45
74. Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., Brunton, G., & Kavanagh, J. (2004). Integrating qualitative research with trials in systematic reviews. British Medical Journal, 328(7446), 1010-1012. doi: 10.1136/bmj.328.7446.1010
75. Torgerson, D.J., & Torgerson, C.J. (2008). Designing randomised controlled trials in health, education and the social sciences: An introduction. Basingstoke: Palgrave Macmillan.
76. US Department of Education. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Office of Planning, Evaluation, and Policy Development. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.doc
77. Ungerleider, C., & Burns, T. (2003). A systematic review of the effectiveness and efficiency of networked ICT in education. A state of the field report to the Council of Ministers of Education, Canada and Industry Canada (Unpublished report). Retrieved from http://www.cmec.ca/Publications/Lists/Publications/Attachments/55/SystematicReview2003.en.pdf
78. Waight, C.L., Willging, P., & Wentling, T.L. (2004) Recurrent themes in e-learning: A narrative analysis of major e-learning reports. Quarterly Review of Distance Education, 5(3), 195-203. Retrieved from http://www.infoagepub.com/Quarterly-Review-of-Distance-Education.html
79. Walsh, D. & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Journal of Advanced Nursing, 50(2), 204-211. doi: 10.1111/j.1365-2648.2005.03380.x
80. Weber, R.P. (1990). Basic content analysis (2nd ed.). Newbury Park, CA: Sage.
81. What Works Clearinghouse. (2008). WWC procedures and standards handbook (version 2.0). Princeton, NJ: US Department of Education, Institute of Education Sciences. Retrieved from http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=19&tocid=1
82. Williams, S.L. (2006). The effectiveness of distance education in allied health science programs: A meta-analysis of outcomes. American Journal of Distance Education, 20(3), 127-141. doi: 10.1207/s15389286ajde2003_2
83. Wong, G., Greenhalgh, T., & Pawson, R. (2010). Internet-based medical education: A realist review of what works, for whom and in what circumstances. BMC Medical Education, 10(1), 12. Retrieved from http://www.biomedcentral.com/1472-6920/10/12
84. Yin, R.K., & Heald, K.A. (1975). Using the case survey method to analyze policy studies. Administrative Science Quarterly, 20(3), 371-381. Retrieved from http://www.johnsoncornell.edu/publications/asq/
85. Zhao Y., Lei, J., Lai, B.Y.C., & Tan, H.S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107(8), 1836-1884. doi: 10.1111/j.1467-9620.2005.00544.x