A framework for evaluating the quality of multimedia learning resources

Educational Technology and Society

Volumn 10, Issue 2, 2007, Pages 44-59

  Leacock, Tracey L ^a   Nesbit, John C ^a  

^a SIMON FRASER UNIVERSITY   (Canada)

Author keywords

eLera; Learning resource; LORI; Learning object; Quality

Indexed keywords

EID: 34250882617     PISSN: 11763647     EISSN: 14364522     Source Type: Journal    
DOI: None     Document Type: Review

References (123)

1. Advanced Distributed Learning (2003). SCORM Overview. Retrieved July 26, 2006, from http://www.adlnet.org/index.cfm?fuseaction=scormabt.
2. Aleven, V., Stahl, E., Schworm, S., Fischer, F., & Wallace, R. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research, 73(3), 277-320.
3. Ananda, S. (2003). Achieving alignment. Leadership, September/October, 18-21 & 37.
4. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4, 167-207.
5. A-Prompt: Web accessibility verifier (n.d.). Retrieved June 27, 2006, from http://aprompt.snow.utoronto.ca/
6. Bangert-Drowns, R. L., Kulik, C-L. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effects of feedback in test-like events. Review of Educational Research, 61, 179-212.
7. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16.
8. Bush, M.D. (2002). Connecting instructional design to international standards for content reusability. Educational Technology, 42(6), 5-13.
9. Campbell, L. M. (2003). Engaging with the learning object economy. In A. Littlejohn (Ed.) Reusing online resources: A sustainable approach to e-learning. London: Kogan-Page, 35-45.
10. Cohen, S. A. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16-20.
11. Dall'Alba, G., Walsh, E., Bowden, J., Martin, E., Masters, G., Ramsden, P., & Stephanou, A. (1993). Textbook treatments and students' understanding of acceleration. Journal of Research in Science Teaching, 30(7), 621-635.
12. De Laurentiis, E. C. (1993). How to recognize excellent educational software. New York: Lifelong Software.
13. Du Boulay, B. & Luckin, R. (2001) Modelling human teaching tactics and strategies for tutoring systems. International Journal of Artificial Intelligence in Education, 12(3), 235-256. Retrieved July 26, 2006, from http://www.informatics.sussex.ac.uk/users/bend/papers/ijaiedteachers.pdf.
14. Dublin Core Metadata Initiative (1999). Dublin Core Metadata Element Set, Version 1.1: Reference Description. Retrieved July 26, 2006, from http://dublincore.org/documents/1999/07/02/dces/.
15. Duval, E. & Hodgins, W. (2006). Standardized uniqueness: Oxymoron or vision for the future? Computer, 39(3), 96-98.
16. Eom, W. Y., & Reiser, R. A. (2000). The effects of self-regulation and instructional control on performance and motivation in computer-based instruction. International Journal of Instructional Media, 27(3), 247-260.
17. Fleming, M., & Levie, W. H. (Eds.). (1998). Instructional message design: Principles from the behavioral and cognitive sciences (2nd ed.). Englewood Cliffs, NJ: Educational Technology Publications.
18. Friesen, N. & Fisher, S. (2003). CanCore Guidelines Version 2.0: Introduction. Retrieved March 17, 2005, from http://www.cancore.ca/ guidelines/CanCore_Guidelines_Introduction_2.0.pdf.
19. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation and learning: A research and practice model. Simulation and Gaming, 33(4), 441-467.
20. Golden, D. C. (2002). Instructional software accessibility: A status report. Journal of Special Education and Technology, 17(1), 57-60.
21. Goldstein, I.P. (1982). The genetic graph: A representation for the evolution of procedural knowledge. In D. Sleeman & J. S. Brown (Eds.) Intelligent tutoring systems (pp. 51-77). New York: Academic Press.
22. Gollnick, D., & Chinn, P. C. (1991). Multicultural education for exceptional children. ERIC digest no. E498. Reston, VA: ERIC Clearinghouse on Handicapped and Gifted Children. ED 333 620.
23. Grice, H. P. (1975). Logic and conversation. In P. Cole and J. L. Morgan (Eds.) Syntax and Semantics Vol. 3: Speech Acts (41-58). New York: Seminar Press.
24. Guzdial, M. (1994). Software-realized scaffolding to facilitate programming for science learning. Interactive Learning Environments, 4, 1-44.
25. Hadwin, A. F., & Winne, P. H. (2001). CoNoteS2: A software tool for promoting self-regulation and collaboration. Educational Research and Evaluation: An International Journal on Theory and Practice, 7, 313-34.
26. Hall, R, H., Hall, M. A., & Saling, C. B. (1999). The effects of graphical postorganization strategies on learning from knowledge maps. Journal of Experimental Education, 67, 101-112.
27. Hamel, C. J. & Ryan-Jones, D. (2002). Designing instruction with learning objects. International Journal of Educational Technology, 3(1).
28. Harden, R. M. (2005). A new vision for distance learning and continuing medical education. The Journal of Continuing Education in the Health Professions, 25, 43-51.
29. Haughey, M., & Muirhead, B. (2005). Evaluating learning objects for schools. E-Journal of Instructional Science and Technology, 8(1). Retrieved June 27, 2006, from http://www.usq.edu.au/electpub/ejist/docs/ vol8_no1/content.htm.
30. Hedberg, J. (2004). Designing multimedia: Seven discourses. Cambridge Journal of Education, 34, 241-256.
31. Hill, J. R. & Hannafin, M. J. (2001). Teaching and learning in digital environments: The resurgence of resource-based learning. Educational Technology Research and Development 49(3), 37-52.
32. Hirumi, A. (2005). In search of quality: Analysis of e-learning guidelines and specifications. The Quarterly Review of Distance Education, 6, 309-330.
33. Hodson, D. (2001). What counts as good science education? OISE Papers in STSE Education, 2, 7-22. Retrieved August 30, 2005, from www.sccao.org/downloads/derek%20oisepapers.pdf.
34. IEEE (2002). IEEE Learning Object Metadata. Retrieved August 30, 2005, from http://ltsc.ieee.org/wg12/index.html.
35. IMS (2005). IMS Global Learning Consortium Specifications. Retrieved July 26, 2006, from http://www.imsglobal.org/specifications.cfm.
36. IMS (2002). IMS guidelines for developing accessible learning applications, version 1.0. IMS Global Learning Consortium. Retrieved July 26, 2006, from http://www.imsglobal.org/accessibility.
37. IMS (2006). Tools interoperability guidelines. Retrieved November 20, 2006, from http://www.imsglobal.org/ti/index.html.
38. Jackson, S. L., Krajcik, J., & Soloway, E. (1998). The design of guided learner-adaptable scaffolding in interactive learning environments. CHI 98, Los Angeles, CA.
39. Jones, A., Seanlon, C., Tosunoglu, E., Morris, S., Ross, S., Butcher, P., & Greenberg, J. (1999). Contexts for evaluating educational software. Interacting with Computers, 11, 499-516.
40. Kalyanpur M. & Kirmani, M. H. (2005). Diversity and technology: Classroom implications of the digital divide. Journal of Special Education Technology 20(4), 9-18.
41. Kearsley, G. (1988). Online help systems: Design and implementation. Norwood, NJ: Alex Publications.
42. Keller, J. M. (1987). Strategies for stimulating the motivation to learn. Performance and Instruction, 26(8), 1-7.
43. Keller, J. M. & Suzuki, K. (2004). Learner motivation and e-learning design: A multinationally validated process. Journal of Educational Media, 29(3), 229-239.
44. Koppi, T., Bogle, L. & Bogle, M. (2005). Learning objects, repositories, sharing and reusability. Open Learning, 20(1), 83-91.
45. Krull, G. E., Mallinson, B. J., & Sewry, D. A. (2006). Describing online learning content to facilitate resource discovery and sharing: The development of the RU LOM Core. Journal of Computer Assisted Learning, 22, 172-181.
46. Lambiotte, J. G., Dansereau, D. F., Cross, D. R., & Reynolds, S. B. (1989). Multirelational semantic maps. Educational Psychology Review, 1, 331-367.
47. Laurillard, D., Stratfold, M., Luckin, R., Plowman, L. & Taylor, J. (2000). Affordances for learning in a nonlinear narrative medium. Journal of Interactive Media in Education, 2 Retrieved August 30, 2005, from http://www-jime.open.ac.uk/00/2.
48. Leacock, T., Richards, G., & Nesbit, J. (2004). Teachers need simple, effective tools to evaluate learning objects: Enter eLera. Proceedings of the IASTED International Conference on Computers and Advanced Technology in Education, 7, 333-338.
49. Li, J. Z., Nesbit, J. C., & Richards, G. (2006). Evaluating learning objects across boundaries: The semantics of localization. Journal of Distance Education Technologies, 4, 17-30.
50. Liebling, C. R. (1997) Achieving standards-based curriculum alignment through mindful teaching. Retrieved August 30, 2005, from r3cc.ceee.gwu.edu/teaching_learning/Curalgn3.pdf.
51. Liu, L. & Johnson, D. L. (2005). Web-based resources and applications: Quality and influence. Computers in the Schools, 21, 131-146.
52. Lohr, L. L. (2000). Designing the instructional interface. Computers in Human Behavior, 16, 161-182.
53. Luckin, R. & du Boulay, B. (1999). Ecolab: The development and evaluation of a Vygotskian design framework. International Journal of Artificial Intelligence in Education, 10, 198-220.
54. Malloy, T. E., & Hanley, G. L. (2001). MERLOT: A faculty-focused Web site of educational resources, Behavior Research Methods, Instruments, & Computers, 33, 274-276.
55. Mandel, T (2002). Quality technical information: Paving the way for usable print and web interface design. ACM Journal of Computer Documentation, 26(3), 118-125.
56. Mardis, M. A. & Zia, L. L. (2003). Leading the wave of science and mathematics learning innovation: NSDL resources. Knowledge Quest, 31(3), 10-11.
57. Martens, R. L. Gulikers, J. & Bastiaens, T. (2004). The impact on intrinsic motivation on e-Learning in authentic computer tasks. Journal of Computer Assisted Learning, 20, 368-376.
58. Mayer, R. E. & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43-52.
59. Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
60. Mayes, J. T. & Fowler, C. J. (1999). Learning technology and usability: A framework for understanding courseware. Interacting with Computers, 11, 485-497.
61. Mayhew, D. J. (1992). Principles and guidelines in software user interface design. Englewood Cliffs, NJ: Prentice Hall.
62. McClelland, M. (2003). Metadata standards for educational resources. Compute, 36(11), 107-109.
63. McKendree, J. E. (1990). Effective feedback content for tutoring complex skills. Human Computer Interaction, 5, 381-414.
64. MERLOT (2000). Evaluation criteria for peer reviews of MERLOT learning resources. Retrieved July 26, 2006, from http://taste.merlot.org/catalog/ peer_review/eval_criteria.htm.
65. Metros, S. E. (2005). Visualizing knowledge in new educational environments: A course on learning objects. Open Learning, 20(1), 93-102.
66. nd IARTEM volume (pp. 121-140). Stockholm: Stockholm Institute of Education Press.
67. Morrison, G., Ross, S. M., Gopalakrishna, M., & Casey, J. (1995). The effects of feedback and incentives on achievement in computer-based instruction. Contemporary Educational Psychology, 20, 32-50.
68. Nesbit, J. C. & Adesope, O. O. (2006). Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research, 76, 413-448.
69. Nesbit, J. C. & Belfer, K. (2004). Collaborative evaluation of learning objects. In R. McGreal (Ed.) Online education using learning objects (138-153). New York: RoutledgeFalmer.
70. Nesbit, J. C., Belfer, K., & Leacock T. L. (2004) LORI 1.5: Learning Object Review Instrument. Retrieved July 26, 2006, from http://www.elera.net.
71. Nesbit, J. C., Belfer, K., & Vargo, J. (2002). A convergent participation model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3), 105-120.
72. Nesbit, J. C., Li, J., & Leacock, T. L. (2006). Web-based tools for collaborative evaluation of learning resources. Retrieved November 9, 2006, from http://www.elera.net/eLera/Home/Articles/WTCELR.
73. Nesbit, J., Leacock, T., Xin, C., & Richards, G. (2004). Learning Object Evaluation and Convergent Participation: Tools for Professional Development in E-Learning in Computers and Advanced Technology in Education: Proceedings of the Seventh IASTED International Conference. Hawaii: ACTA, 339-344.
74. Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Proceedings of the Association for Computing Machinery CHI 94 Conference, Boston, MA, 152-158.
75. Nielsen, J. (1997). The need for speed. Jakob Nielsen's Alertbox. Retrieved March 15, 2005, from http://www.useit.com/alertbox/9703a.html.
76. Norman, D. (1988). The design of everyday things. New York: Basic Books.
77. Paciello, M. (2000). Web Accessibility for people with disabilities. Norwood, MA: CMP Books.
78. Palmeri, J. (2006). Disability studies, cultural practice, and the critical practice of technical communication pedagogy. Technical Communication Quarterly, 15(1), 49-65.
79. Park, O-C. (1996). Adaptive instructional systems. In D. H. Jonassen (Ed.) Handbook of research for educational communications and technology (pp. 138-153). New York: Macmillan.
80. Parlangeli, O., Marchigiani, E., & Bagnara, S. (1999). Multimedia systems in distance education: Effects of usability on learning. Interacting with Computers, 12, 37-49.
81. Parrish, P.E. (2004). The trouble with learning objects. Educational Technology Research and Development, 52(1), 49-67.
82. Pearson, R. & van Schaik, P. (2003). The effect of spatial layout and link colour of web pages on performance in a visual search task and an interactive search task. International Journal of Human-Computer Studies, 59, 327-353.
83. Peterson-Karlan, G. R. & Parette, P. (2005). Millenial students with mild disabilities and emerging assistive technology trends. Journal of Special Education Technology 20(4), 27-38.
84. Porter, A. C. (2002). Measuring the content of instruction: Uses in research practice. Educational Researcher, 31(7), 3-14.
85. Ramsay, J., Barbesi, A., & Preece, J. (1998). A psychological investigation of long retrieval times on the World Wide Web. Interacting with Computers, 10, 77-86.
86. Reed, S. K. (2006). Cognitive architectures for multimedia learning. Educational Psychologist, 41, 87-98.
87. Richards, G. & Nesbit, J. C. (2004). The teaching of quality: Convergent participation for the professional development of learning object designers. International Journal of Technologies in Higher Education, 1(3), 56-63.
88. Roach, A. T., Elliott, S. N., & Webb, N. E. (2000). Alignment of an alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin Alternate Assessment. The Journal of Special Education, 38(4), 218-231.
89. Robson, R. (2004). Context and the role of standards in increasing the value of learning objects. In R. McGreal (Ed.) Online education using learning objects. New York: RoutledgeFalmer.
90. Roschelle J. & DiGiano, C. (2004). ESCOT: Coordinating the influence of R&D and classroom practice to produce educational software from reusable components. Interactive Learning Environments, 12(1-2), 73-107.
91. Rothman, R., Slattery, J. B., Vraneck, J. L., & Resnick, E. B. (2002). Benchmarking and Alignment of Standards and Testing. (Center for the Study of Evaluation Technical Report 566). Los Angeles: National Centre for Research on Evaluation, Standards, and Student Testing.
92. Ryan R. M. & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions, Contemporary Educational Psychology, 25, 54-67.
93. Ryder, M. & Wilson, B. (1996). Affordances and constraints of the Internet for learning and instruction. Presented to a joint session of the Association for Educational Communications Technology. Retrieved July 26, 2006, from http://carbon.cudenver.edu/~mryder/aect_96.html.
94. Sampson, D.G. & Karampiperis, P. (2004). Reusable learning objects: Designing metadata systems supporting interoperable learning object repositories. In R. McGreal (Ed.) Online education using learning objects (pp. 207-221). New York: RoutledgeFalmer.
95. Sanger, M. J. & Greenbowe, T. J. (1999). An analysis of college chemistry textbooks as sources of misconceptions and errors in electrochemistry. Journal of Chemistry Education, 76(6), 853-860.
96. Schraw, G., Flowerday, T., & Lehman, S. (2001). Increasing situational interest in the classroom. Educational Psychology Review, 13 (3), 211-224.
97. Section 508 Standards (1998). Retrieved July 26, 2006, from http://www.section508.gov.
98. Selvidge, P., Chaparro, B., & Bender, G. (2001). The world wide wait: Effects of delays on user performance. International Journal of Industrial Ergonomics, 29(1), 15-20.
99. Shavinina, L. V. & Loarer, E. (1999). Psychological evaluation of educational multimedia applications. European Psychologist, 4(1), 33-44.
100. SimCity (n.d.). Tips & tricks. Retrieved March 10, 2005, from http://simcity.ea.com/tipstricks/tipsntricks.php.
101. Small, R. (1997). Assessing the motivational qualities of world wide web sites. US Department of Education Report.
102. Southern Regional Educational Board (2005). Principles of effective learning objects: Guidelines for development and use of learning objects for the SCORE initiative of the Southern Regional Educational Board. Retrieved October 15, 2006, from http://sreb.org/programs/EdTech/pubs/PDF/05T03- PrinciplesEffectiveLO.pdf.
103. Squires, D. & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11, 467-483.
104. Strunk, W., Osgood, V., & Angell, R. (2000). The Elements of Style, (4th ed.). Boston: Allyn & Bacon.
105. Tennyson, R. D. & Christensen, D.L. (1988). MAIS: An intelligent learning system. In D. Jonassen (Ed.) Intelligent designs for microcomputer courseware (pp. 247-274). Hillsdale, NJ: Erlbaum.
106. Tognazzini, B. (2001). First Principles. AskTOG, Nielsen Norman Group. Retrieved July 26, 2006, from http://www.asktog.com/basics/firstPrinciples.html.
107. Treviranus, J. & Brewer, J. (2003). Developing and reusing accessible content and applications. In A. Littlejohn (Ed.) Reusing online resources: A sustainable approach to e-learning (pp. 119-128). London: Kogan-Page.
108. Tselios, N. K., Avouris, N. M., Dimitracopoulou, A., & Daskakaki, S. (2001). Evaluation of distance-learning environments: Impact of usability in student performance. International Journal of Educational Telecommunications, 7, 355-378.
109. Tsui, C-Y., & Treagust, D. F. (2004). Motivational aspects of learning genetics with interactive multimedia. The American Biology Teacher, 66 (4), 277-286.
110. Tufte, E. R. (1997). Visual explanations: Images and quantities, evidence and narrative. Cheshire, CN: Graphics Press.
111. UsableNet.com (2004). Retrieved June 27, 2006, from http://www.usablenet. com/.
112. van Merrienboer, J. J. G. & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17(2), 147-177.
113. Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25(3), 198-205.
114. Vekiri, I. (2002). What is the value of graphical displays in learning? Educational Psychology Review, 14(3), 261-312.
115. Wainer, H. (2000). Introduction and history. In H. Wainer (Ed.) Computerized adaptive testing: A primer, (2nd ed.) (pp. 1-22). Hillsdale, NJ: Erlbaum.
116. Webb, N. L., Alt, M., Cormier, R. E., & Vesperman, B. (2006). The web alignment tool: Development, refinement, and dissemination. In Aligning assessment to guide the learning of all students. Washington, DC: Council of Chief State School Officers Report, 1-30.
117. WebXACT (2004). Retrieved, June 27, 2006, from http://webxact.watchfire. com/.
118. nd ed). Upper Saddle River, NJ: Prentice Hall.
119. Wigfield, A. & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68-81.
120. Williams, M. D. (1996). Learner-control and instructional technologies. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (957-983). New York: Simon & Schuster Macmillan.
121. World Wide Web Consortium (2006). W3C technical reports and publications. Retrieved November 20, 2006, from http://www.w3.org/TR/.
122. World Wide Web Consortium (1999). Web Content Accessibility Guidelines 1.0, Retrieved July 26, 2006, from http://www.w3.org/TR/WCAG10.
123. Zimmerman, B. J. (1998). Self-fulfilling cycles of academic regulation. In D. H. Schunk and B. J. Zimmerman (Eds.). Self-regulated learning: From teaching to self-reflective practice (1-19). New York: Guilford.