Meeting the Challenges in Evaluating Mobile Learning: A 3-Level Evaluation Framework

International Journal of Mobile and Blended Learning (IJMBL)

Volumn 1, Issue 2, 2009, Pages 54-75

  Vavoula, Giasemi ^a   Sharples, Mike ^b  

^a UNIVERSITY OF LEICESTER   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

Author keywords

Ethics; Evaluation Framework; Informality and Formality of Learning; Learning Context; Learning Outcomes; Mobile Learning Evaluation; Requirements for Evaluation

Indexed keywords

EID: 85001790674     PISSN: 19418647     EISSN: 19418655     Source Type: Journal    
DOI: 10.4018/jmbl.2009040104     Document Type: Article

References (50)

1. Ahonen, M. (2008). Sustainable Wireless Computing - How Public Information Systems Can Be Designed to Reduce Exposure. In Proceedings of 31st IRIS Scandinavian Information Systems Conference, Åre, Sweden
2. Anastopoulou, S., Sharples, M., Wright, M., Martin, H., Ainsworth, S., Benford, S., Crook, C., Greenhalgh, C., & O'Malley, C. (2008). Learning 21st Century Science in Context with Mobile Technologies. In Traxler, J., Riordan, B. and Dennett, C. (Eds.), Proceedings of mLearn 2008: The bridge from text to context, Wolverhampton, UK, University of Wolverhampton, (pp. 12–19).
3. Bakardjieva, M., Feenberg, A., & Goldie, J. (2004). User-Centered Internet Research: The Ethical Challenge. In Buchanan, E.A. (Ed.), Readings in Virtual Research Ethics: Issues and Controversies, Idea Group Inc (IGI), (pp. 338–350).
4. Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, G., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R.C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for Agile Software Development. Available online: www.agilemanifesto.org. Accessed on 22/01/09.
5. BECTA (2008). Harnessing Technology: Next Generation Learning 2008–14. Available online: http://publications.becta.org.uk/display.cfm?resID=38751&page=1835.
6. Black, P., & Wiliam, D. (1998a). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74.
7. Black, P. & Wiliam, D. (1998b). Inside the black box: raising standards through classroom assessment. Phi Delta Kappan, 80, 139–148.
8. Boud, D. (1995). Assessment and learning: contradictory or complementary? In P. Knight (Ed.), Assessment for Learning in Higher Education. London, Kogan Page, (pp. 35–48
9. Buchanan, E. A. (Ed.) (2004). Readings in Virtual Research Ethics: Issues and Controversies. Idea Group Inc (IGI).
10. Clough, G., & Jones, A. (2006). The Uses of PDAs and Smartphones in Informal learning. In Proceedings of MLearn 2006, Banff, Canada.
11. Colley, H., Hodkinson, P., & Malcom, J. (2003). Informality and formality in learning. Learning and Skills Research Centre. 1492/11/03/500.
12. Corlett, D., Sharples, M., Chan, T., & Bull, S. (2005). Evaluation of a Mobile Learning Organiser for University Students. Journal of Computer Assisted Learning, 21, 162–170.
13. Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327–358.
14. Fountain, J. E. (2001). Building the Virtual State: Information Technology and Institutional Change, Brookings Institution Press.
15. Greenhalgh, C., French, A., Tennent, P., Humble, J., & Crabtree, A. (2007). From Replay Tool to Digital Replay System. In Proceedings of e-Social Science Conference, Ann Arbor, Michigan, USA.
16. Griffin, J., & Symington, D. (1998). Finding Evidence of Learning in Museum Settings. In Proceedings of Evaluation and Visitor Research Special Interest Group Conference ‘Visitors Centre Stage: Action for the Future’, Canberra.
17. Hagen, P., Robertson, T., Kan, M., & Sadler, K. (2005). Emerging research methods for understanding mobile technology use. In Proceedings of CHISIG, Australia, (pp 1–10).
18. Hidi, S., & Harackiewicz, J.M. (2000). Motivating the Academically Unmotivated: A Critical Issue for the 21st Centrury. Review of Educational Research, 70(2), 151–179.
19. Hsi, S. (2007). Conceptualizing Learning from the Everyday Activities of Digital Kids. International Journal of Science Education, 29(12), 1509 - 1529.
20. Kaplan, R. S., & Norton, D. P. (1996). Using the balanced scorecard as a strategic management system. Harvard Business Review, (pp. 75–85).
21. Kjeldskov, J., & Stage, J. (2004). New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human-Computer Studies, 60, (pp. 599–620).
22. Knight, P. (2001). A Briefing on Key Concepts: Formative and summative, criterion and norm-referenced assessment. LTSN Generic Centre.
23. Larman, C., & Basili, V.R. (2003). Iterative and Incremental Development: A Brief History. Computer Supported Cooperative Work, 36(6), 47–56.
24. Meek, J. (2006). Adopting a Lifecycle Approach to the Evaluation of Computers and Information Technology. PhD Thesis, School of Electronic, Electrical and Computer Engineering, The University of Birmingham.
25. Moore, R. W., & Sutman, F. X. (1970). The development, field test and validation of an inventory of scientific attitudes. Journal of Research in Science Teaching, 7, 85–94.
26. Oliver, M., & Harvey, J. (2002). What does ‘impact’ mean in the evaluation of learning technology? Educational Technology & Society, 5(3), 18–26.
27. Pande, P. S., Neuman, R. P., & Cavanagh, R. R. (2000). The Six Sigma Way: How GE, Motorola, and Other Top Companies are Honing Their Performance. McGraw-Hill.
28. Papadimitriou, I., Tselios, N., & Komis, V. (2007). Analysis of an informal mobile learning activity based on activity theory. In Vavoula, G.N., Kukulska-Hulme, A. and Pachler, N., Proceedings of Workshop Research Methods in Informal and Mobile Learning, WLE Centre, Institute of Education, London, UK, (pp. 25–28).
29. Patrick, K., Griswold, W., Raab, F. & Intille, S. (2008). Health and the Mobile Phone. American Journal of Preventive Medicine, 35(2), 177–181.
30. Price, S., & Oliver, M. (2007). A Framework for Conceptualising the Impact of Technology on Teaching and Learning. Educational Technology & Society, 10(1), 16–27.
31. Riley, D. (2007). Educational Technology and Practice: Types and Timescales of Change. Educational Technology & Society, 10, 85–93.
32. Roto, V., Oulasvirta, A., Haikarainen, T., Kuorelahti, J., Lehmuskallio, H. & Nyyssonen, T. (2004). Examining mobile phone use in the wild with quasi-experimentation. HIIT Technical Report 2004–1. Available online: www.hiit.fi/publications.
33. Royce, W. W. (1970). Managing the Development of Large Software Systems. In Proceedings of IEEE WESCON, (pp. 1–9).
34. Ryu, H., & Parsons, D. (2008). Designing Learning Activities with Mobile Technologies. In Ryu, H. and Parsons, D. (Eds.), Innovative Mobile Learning: Techniques and Technologies, Idea Group Inc (IGI), (pp. 1–20).
35. Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler & R. M. Gagne, & M. Scriven (Eds.), Perspectives of Curriculum Evaluation. Chicago, Rand McNally, (pp. 39–83).
36. Sharples, M., Taylor, J., & Vavoula, G. (2007a). A Theory of Learning for the Mobile Age. In Andrews, R. and Haythornthwaite, C. (Eds.), The Sage Handbook of E-learning Research. London, Sage, (pp. 221–247).
37. Sharples, M., Lonsdale, P., Meek, J., Rudman, P. D., & Vavoula, G. N. (2007b). An Evaluation of MyArtSpace: a Mobile Learning Service for School Museum Trips. In Proceedings of mLearn 2007, Melbourne, Australia.
38. Sharples, M., Jeffery, N., Du Boulay, J. B. H., Teather, D., Teather, B., & Du Boulay, G. H. (2002). Socio-cognitive engineering: a methodology for the design of humancentred technology. European Journal of Operational Research, 136(2), 310–323.
39. Smith, H., Heldt, S., Fitzpatrick, G., Hui Ng, K., Benford, S., Wyeth, P., Walker, K., Underwood, J., Luckin, R., & Good, J. (2007). Reconstructing an informal mobile learning experience with multiple data streams. In Vavoula, G.N., Kukulska-Hulme, A. and Pachler, N., Proceedings of Workshop Research Methods in Informal and Mobile Learning, WLE Centre, Institute of Education, London, UK., (pp. 29–34).
40. Taylor, J. (2004). A task-centred approach to evaluating a mobile learning environment for pedagogical soundness. In Proceedings of MLearn2004, London, UK, Learning and Skills Development Agency, (pp. 167–171).
41. Taylor, J., Sharples, M., O'Malley, C., Vavoula, G., & Waycott, J. (2006). Towards a Task Model for Mobile Learning: a Dialectical Approach. International Journal of Learning Technology, 2(2/3), (pp. 138–158).
42. Traxler, J. (forthcoming). Mobile Learning Evaluation: The Challenge of Mobile Societies. In G. Vavoula, N. Pachler, & A. Kukulska-Hulme (Eds.), Researching Mobile Learning: Frameworks, Methods and Research Designs. Oxford, Peter Lang.
43. Traxler, J., & Bridges, N. (2004). Mobile Learning - The Ethical and Legal Challenges. In Proceedings of MLEARN 2004, Bracciano, Italy.
44. Traxler, J., & Kukulska-Hulme, A. (2005). Evaluating Mobile Learning: Reflections on Current Practice. In Proceedings of MLEARN 2005, Cape Town, South Africa.
45. Trinder, J., Roy, S., & Magill, J. (2007). Have You Got Your PDA With You?. Denials and Accusations. In G. N.Vavoula, A. Kukulska-Hulme, & N. Pachler, (Eds.), Proceedings of Workshop Research Methods in Informal and Mobile Learning, Institute of Education, London, UK.
46. Vavoula, G., Meek, J., Sharples, M., Lonsdale, P. & Rudman, P. (2006a). A lifecycle approach to evaluating MyArtSpace. In S. Hsi, Kinshuk, T. Chan, & D. Sampson (Eds.), Proceedings of 4th International Workshop of Wireless, Mobile and Ubiquitous Technologies in Education (WMUTE 2006), Athens, Greece, IEEE Computer Society, (pp. 18–22)
47. Vavoula, G., Sharples, M., Rudman, P., Lonsdale, P., & Meek, J. (2007). Learning Bridges: a role for mobile technologies in education. Educational Technology Magazine, XLVII(3), 33–37.
48. Vavoula, G., Sharples, M., Rudman, P., Meek, J., & Lonsdale, P. (forthcoming). Myartspace: Design and evaluation of support for learning with multimedia phones between classrooms and museums. Computers and Education.
49. Vavoula, G., Sharples, M., Rudman, P., Meek, J., Lonsdale, P., & Philips, D. (2006b). Museum explorations in physical, personal and virtual space through MyArtSpace. In Proceedings of mLearn 2006, Banff, Canada.
50. Vavoula, G. N. (2005). WP4: A Study of Mobile Learning Practices. MOBIlearn deliverable D4.4. Available online: http://www.mobilearn.org/download/results/public_deliverables/MOBIlearn_D4.4_Final.pdf’