Instructional design variations in internet-based learning for health professions education: A systematic review and meta-analysis

Academic Medicine

Volumn 85, Issue 5, 2010, Pages 909-922

  Cook, David A ^a   Levinson, Anthony J ^b   Garside, Sarah ^b   Dupras, Denise M ^a   Erwin, Patricia J ^a   Montori, Victor M ^a  

^a MAYO CLINIC   (United States)

^b MCMASTER UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77953585957     PISSN: 10402446     EISSN: None     Source Type: Journal    
DOI: 10.1097/ACM.0b013e3181d6c319     Document Type: Article

References (93)

1. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81:207-212.
2. Cook DA. Web-based learning: Pros, cons, and controversies. Clin Med. 2007;7:37-42.
3. Association of American Medical Colleges Institute for Improving Medical Education. Effective Use of Educational Technology in Medical Education: Colloquium on Educational Technology: Recommendations and Guidelines for Medical Educators. Available at: https://services.aamc.org/ publications/showfile.cfm? file-version80. pdf&prd-id-184&prv-id-224&pdf-id-80. Accessed January 12, 2010.
4. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: A metaanalysis. JAMA. 2008;300:1181-1196.
5. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: Sound educational method or hype? A review of the evaluation literature. Acad Med. 2002;77(10 suppl):S86-S93.
6. Tallent-Runnels MK, Thomas JA, Lan WY, Cooper S. Teaching courses online: A review of the research. Rev Educ Res. 2006;76:93-135.
7. Cohen PA, Dacanay LD. Computer-based instruction and health professions education: A meta-analysis of outcomes. Eval Health Prof. 1992;15:259-281.
8. Cook DA. Where are we with Web-based learning in medical education? Med Teach. 2006;28:594-598.
9. Clark RE. Reconsidering research on learning from media. Rev Educ Res. 1983;53:445-459.
10. Keane DR, Norman GR, Vickers J. The inadequacy of recent research on computerassisted instruction. Acad Med. 1991;66:444-448.
11. Friedman CP. The research we should be doing. Acad Med. 1994;69:455-457.
12. Tegtmeyer K, Ibsen L, Goldstein B. Computer-assisted learning in critical care: From ENIAC to HAL. Crit Care Med. 2001; 29(8 suppl):N177-N182.
13. Cook DA. The research we still are not doing: An agenda for the study of computer-based learning. Acad Med. 2005;80:541-548.
14. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896-1900.
15. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: A proposal for Reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) Group. JAMA. 2000;283:2008-2012.
16. Kirkpatrick D. Revisiting Kirkpatrick's fourlevel model. Training Dev. 1996;50:54-59.
17. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86:420-428.
18. Wells GA, Shea B, O'Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Available at: http://www.ohri. ca/programs/clinical-epidemiology/oxford. htm. Accessed January 12, 2010.
19. Morris SB, DeShon RP. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol Methods. 2002;7:105-125.
20. Dunlap WP, Cortina JM, Vaslow JB, Burke MJ. Meta-analysis of experiments with matched groups or repeated measures designs. Psychol Methods. 1996;1:170-177.
21. Hunter JE, Schmidt FL. Methods of Metaanalysis: Correcting Error and Bias in Research Findings. Thousand Oaks, Calif: Sage; 2004.
22. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions 5.0.0. Available at: http://www. cochrane.org/resources/handbook/ index.htm. Accessed May 29, 2008.
23. Rosenthal R. Parametric measures of effect size. In: Cooper HM, Hedges LV, eds. The Handbook of Research Synthesis. New York, NY: Russell Sage Foundation; 1994:231-244.
24. Curtin F, Altman DG, Elbourne D. Metaanalysis combining parallel and cross-over clinical trials. I: Continuous outcomes. Stat Med. 2002;21:2131-2144. (Pubitemid 34830752)
25. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in metaanalyses. BMJ. 2003;327:557-560.
26. Kaplan IP, Patton LR, Hamilton RA. Adaptation of different computerized methods of distance learning to an external pharmd degree program. Am J Pharm Educ. 1996;60:422-425.
27. Chan DH, Leclair K, Kaczorowski J. Problembased small-group learning via the Internet among community family physicians: A randomized controlled trial. MD Comput. 1999;16:54-58.
28. Papa FJ, Aldrich D, Schumacker RE. The effects of immediate online feedback upon diagnostic performance. Acad Med. 1999; 74(10 suppl):S16-S18.
29. Schaad DC, Walker EA, Wolf FM, Brock DM, Thielke SM, Oberg L. Evaluating the serial migration of an existing required course to the World Wide Web. Acad Med. 1999;74(10 suppl):S84-S86.
30. Scherly D, Roux L, Dillenbourg P. Evaluation of hypertext in an activity learning environment. J Comput Assist Learning. 2000;16:125-136.
31. Bearman M, Cesnik B, Liddell M. Random comparison of 'virtual patient' models in the context of teaching clinical communication skills. Med Educ. 2001;35:824-832.
32. Fox N, O'Rourke A, Roberts C, Walker J. Change management in primary care: Design and evaluation of an Internetdelivered course. Med Educ. 2001;35:803-805.
33. Duffy T, Gilbert I, Kennedy D, Wa KP. Comparing distance education and conventional education: Observations from a comparative study of post-registration nurses. Assoc Learning Technol J. 2002;10:70-82.
34. Jedlicka JS, Brown SW, Bunch AE, Jaffe LE. A comparison of distance education instructional methods in occupational therapy. J Allied Health. 2002;31:247-251.
35. Carpenter KM, Watson JM, Raffety B, Chabal C. Teaching brief interventions for smoking cessation via an interactive computer-based tutorial. J Health Psychol. 2003;8:149-160.
36. Chao LW, Enokihara MY, Silveira PS, Gomes SR, Böhm GM. Telemedicine model for training non-medical persons in the early recognition of melanoma. J Telemed Telecare. 2003;9(suppl 1):S4-S7.
37. Frith KH, Kee CC. The effect of communication on nursing student outcomes in a Web-based course. J Nurs Educ. 2003;42: 350-358.
38. Lemaire ED, Greene G. A comparison between three electronic media and in-person learning for continuing education in physical rehabilitation. J Telemed Telecare. 2003;9: 17-22.
39. Maag M. The effectiveness of an interactive multimedia learning tool on nursing students' math knowledge and self-efficacy. Comput Inform Nurs. 2004;22:26-33.
40. Mattheos N, Nattestad A, Christersson C, Jansson H, Attström R. The effects of an interactive software application on the selfassessment ability of dental students. Eur J Dent Educ. 2004;8:97-104.
41. Spickard A 3rd, Smithers J, Cordray D, Gigante J, Wofford JL. A randomised trial of an online lecture with and without audio. Med Educ. 2004;38:787-790.
42. Virvou M, Alepis E. Mobile versus desktop facilities for an e-learning system: Users' perspective. In: Conference Proceedings- The Institute of Electrical and Electronics Engineers (IEEE) International Conference on Systems, Man and Cybernetics. The Hague, The Netherlands: IEEE; 2004;1:48-52.
43. Allison JJ, Kiefe CI, Wall T, et al. Multicomponent Internet continuing medical education to promote chlamydia screening. Am J Prev Med. 2005;28:285-290.
44. Becker EA, Godwin EM. Methods to improve teaching interdisciplinary teamwork through computer conferencing. J Allied Health. 2005; 34:169-176.
45. Brunetaud JM, Leroy N, Pelayo S, et al. Comparative evaluation of two applications for delivering a multimedia medical course in the French-speaking Virtual Medical University (UMVF). Int J Med Inform. 2005;74:209-212.
46. Mukohara K, Schwartz MD. Electronic delivery of research summaries for academic generalist doctors: A randomised trial of an educational intervention. Med Educ. 2005;39: 402-409.
47. Schittek Janda M, Tani Botticelli A, Mattheos N, et al. Computer-mediated instructional video: A randomised controlled trial comparing a sequential and a segmented instructional video in surgical hand wash. Eur J Dent Educ. 2005;9:53-58.
48. Blackmore C, Tantam D, Van Deurzen E. The role of the eTutor-Evaluating tutor input in a virtual learning community for psychotherapists and psychologists across Europe. Int J Psychother. 2006;10:35-46.
49. Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS. Impact of selfassessment questions and learning styles in Web-based learning: A randomized, controlled, crossover trial. Acad Med. 2006; 81:231-238.
50. Friedl R, Höppler H, Ecard K, et al. Comparative evaluation of multimedia driven, interactive, and case-based teaching in heart surgery. Ann Thorac Surg. 2006;82: 1790-1795.
51. Kemper KJ, Gardiner P, Gobble J, Mitra A, Woods C. Randomized controlled trial comparing four strategies for delivering ecurriculum to health care professionals. BMC Med Educ. 2006;6:2.
52. Kerfoot BP, Conlin PR, McMahon GT. Comparison of delivery modes for online medical education. Med Educ. 2006;40:1137-1138.
53. Little BB, Passmore D, Schullo S. Using synchronous software in Web-based nursing courses. Comput Inform Nurs. 2006;24:317-325.
54. Nicholson DT, Chalk C, Funnell WRJ, Daniel SJ. Can virtual reality improve anatomy education? A randomised controlled study of a computer-generated three-dimensional anatomical ear model. Med Educ. 2006;40: 1081-1087.
55. Romanov K, Nevgi A. Learning outcomes in medical informatics: Comparison of a WebCT course with ordinary Web site learning material. Int J Med Inform. 2006;75: 156-162.
56. Thompson GA, Morrison RG, Holyoak KJ, Clark TK. Evaluation of an online analogical patient simulation program. In: Proceedings of the Nineteenth IEEE International Symposium on Computer-Based Medical Systems. Los Alamitos, Calif: IEEE Computer Society Press; 2006:623-628.
57. Al-Rawi WT, Jacobs R, Hassan BA, Sanderink G, Scarfe WC. Evaluation of Web-based instruction for anatomical interpretation in maxillofacial cone beam computed tomography. Dentomaxillofac Radiol. 2007; 36:459-464.
58. Bednar ED, Hannum WM, Firestone A, Silveira AM, Cox TD, Proffit WR. Application of distance learning to interactive seminar instruction in orthodontic residency programs. Am J Orthod Dentofacial Orthop. 2007;132:586-594.
59. Cook DA, Gelula MH, Dupras DM, Schwartz A. Instructional methods and cognitive and learning styles in Web-based learning: Report of two randomised trials. Med Educ. 2007;41: 897-905.
60. Cook DA, Thompson WG, Thomas KG, Thomas MR. Lack of interaction between sensing-intuitive learning styles and problemfirst versus information-first instruction: A randomized crossover trial. Adv Health Sci Educ Theory Pract. 2009;14:79-90.
61. Hege I, Ropp V, Adler M, et al. Experiences with different integration strategies of casebased e-learning. Med Teach. 2007;29:791-797.
62. Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD. Spaced education improves the retention of clinical knowledge by medical students: A randomised controlled trial. Med Educ. 2007;41:23-31.
63. Kerfoot BP, Baker HE, Koch MO, Connelly D, Joseph DB, Ritchey ML. Randomized, controlled trial of spaced education to urology residents in the United States and Canada. J Urol. 2007;177:1481-1487.
64. Miller KT, Hannum WM, Morley T, Proffit WR. Use of recorded interactive seminars in orthodontic distance education. Am J Orthod Dentofacial Orthop. 2007;132:408-414.
65. Moridani M. Asynchronous video streaming vs. synchronous videoconferencing for teaching a pharmacogenetic pharmacotherapy course. Am J Pharm Educ. 2007;71:16.
66. Ridgway PF, Sheikh A, Sweeney KJ, et al. Surgical e-learning: Validation of multimedia Web-based lectures. Med Educ. 2007;41:168-172.
67. Romanov K, Nevgi A. Do medical students watch video clips in eLearning and do these facilitate learning? Med Teach. 2007;29:484-488.
68. Bergeron BP. Learning & retention in adaptive serious games. Stud Health Technol Inform. 2008;132:26-30.
69. Campbell M, Gibson W, Hall A, Richards D, Callery P. Online vs. face-to-face discussion in a Web-based research methods course for postgraduate nursing students: A quasiexperimental study. Int J Nurs Stud. 2008;45: 750-759.
70. Cook DA, Beckman TJ, Thomas KG, Thompson WG. Introducing resident doctors to complexity in ambulatory medicine. Med Educ. 2008;42:838-848.
71. Cook DA, Beckman TJ, Thomas KG, Thompson WG. Adapting Web-based instruction to residents' knowledge improves learning efficiency: A randomized controlled trial. J Gen Intern Med. 2008;23:985-990.
72. Kerfoot BP. Interactive spaced education versus Web based modules for teaching urology to medical students: A randomized controlled trial. J Urol. 2008;179:2351-2356.
73. Kerfoot BP, Brotschi E. Online spaced education to teach urology to medical students: A multi-institutional randomized trial. Am J Surg. 2009;197:89-95.
74. Kopp V, Stark R, Fischer MR. Fostering diagnostic knowledge through computersupported, case-based worked examples: Effects of erroneous examples and feedback. Med Educ. 2008;42:823-829.
75. Tunuguntla R, Rodriguez O, Ruiz JG, et al. Computer-based animations and static graphics as medical student aids in learning home safety assessment: A randomized controlled trial. Med Teach. 2008;30:815-817.
76. Walker BL, Harrington SS. Computer-based instruction and the Web for delivering injury prevention training. Educ Gerontol. 2008;34: 691-708.
77. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum; 1988.
78. Colliver JA, Kucera K, Verhulst SJ. Metaanalysis of quasi-experimental research: Are systematic narrative reviews indicated? Med Educ. 2008;42:858-865.
79. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333:597-600.
80. Adler MD, Johnson KB. Quantifying the literature of computer-aided instruction in medical education. Acad Med. 2000;75:1025-1028.
81. Greenhalgh T. Computer assisted learning in undergraduate medical education. BMJ. 2001; 322:40-44.
82. Wutoh R, Boren SA, Balas EA. eLearning: A review of Internet-based continuing medical education. J Contin Educ Health Prof. 2004; 24:20-30.
83. Dillon A, Gabbard RB. Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control, and style. Rev Educ Res. 1998;68:322-349.
84. Clark RC, Mayer RE. E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. 2nd ed. San Francisco, Calif: Pfeiffer; 2007.
85. Kulik C-LC, Kulik JA, Shwalb BJ. The effectiveness of computer-based adult education: A meta-analysis. J Educ Comput Res. 1986;2:235-252.
86. Fletcher-Flinn CM, Gravatt B. The efficacy of computer-assisted instruction (CAI): A metaanalysis. J Educ Comput Res. 1995;12:219-242.
87. Lewis MJ, Davies R, Jenkins D, Tait MI. A review of evaluative studies of computerbased learning in nursing education. Nurse Educ Today. 2001;21:26-37.
88. Curran VR, Fleet L. A review of evaluation outcomes of Web-based continuing medical education. Med Educ. 2005;39:561-567.
89. Prystowsky JB, Bordage G. An outcomes research perspective on medical education: The predominance of trainee assessment and satisfaction. Med Educ. 2001;35:331-336.
90. Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79: 955-960.
91. Cook DA, Bowen JL, Gerrity MS, et al. Proposed standards for medical education submissions to the Journal of General Internal Medicine. J Gen Intern Med. 2008; 23:908-913.
92. Shea JA. Mind the gap: Some reasons why medical education research is different from health services research. Med Educ. 2001;35: 319-320.
93. Cook DA. The failure of e-learning research to inform educational practice, and what we can do about it. Med Teach. 2009;31:158-162.