The game-based learning evaluation model (GEM): Measuring the effectiveness of serious games using a standardised method

International Journal of Technology Enhanced Learning

Volumn 7, Issue 4, 2015, Pages 326-345

  Oprins, Esther ^a   Visschedijk, Gillian ^a   Roozeboom, Maartje Bakhuys ^a   Dankbaar, Mary ^b   Trooster, Wim ^c   Schuit, Stephanie C E ^b  

^a Department of Training and Performance Innovations ^*  (Netherlands)

^b ERASMUS MEDICAL CENTER   (Netherlands)

^c WINDESHEIM UNIVERSITY OF APPLIED SCIENCES   (Netherlands)

Author keywords

Applied gaming; Educational games; Educational technology; Effectiveness research; Evaluation framework; Evaluation model; Evidencebased; Game based learning; Serious gaming; Validation

Indexed keywords

EID: 84955489961     PISSN: 17535255     EISSN: 17535263     Source Type: Journal    
DOI: 10.1504/IJTEL.2015.074189     Document Type: Article

References (55)

1. Alliger, G.M. and Janak, E.A. (1989) 'Kirkpatrick's levels of training criteria: thirty years later', Personnel Psychology, Vol. 42, pp.331-342.
2. Alvarez, K., Salas, E. and Garofano, C.M. (2004) 'An integrated model of training evaluation and effectiveness', Human Resource Development Review, Vol. 3, No. 4, pp.385-416.
3. Anderson, L.W. and Krathwohl, D.R. (2001) A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Longman, New York.
4. Appleton, J., Christenson, S.L., Kim, D. and Reschly, A.L. (2006) 'Measuring cognitive and psychological engagement: validation of the student engagement instrument', Journal of School Psychology, Vol. 44, pp.427-445.
5. Bakhuys Roozeboom, M., Visschedijk, G. and Oprins, E. (in press) 'The effectiveness of three serious games measuring generic learning features', British Journal of Educational Technology.
6. Bandura, A. (1997) Self-Efficacy: The Exercise of Control, Freeman, New York.
7. Barnard, L., Lan, W.Y., Yen, M.T., Osland Paton, V. and Lai, S. (2009) 'Measuring self-regulation in online and blended learning environments', Internet and Higher Education, Vol. 12, pp.1-6.
8. Bates, R. (2004) 'A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence', Evaluation and Program Planning, Vol. 27, pp.341-347.
9. Bedwell, W.L., Pavlas, D., Heyne, K., Lazzara, E.H. and Salas, E. (2012) 'Toward a taxonomy linking game attributes to learning: an empirical study', Simulation and Gaming, Vol. 43, No. 6, pp.729-760.
10. Bekebrede, G., Warmelink, H.J.G. and Mayer, I.S. (2011) 'Reviewing the need for gaming in education to accommodate the Net generation', Computers and Education, Vol. 57, No. 2, pp.1521-1529, doi:10.1016/j.compedu.2011.02.010.
11. Cannon-Bowers, J.A., Salas, E., Tannenbaum, S.I. and Mathieu, J.E. (1995) 'Toward theoretically based principles of training effectiveness: a model and initial empirical investigation', Military Psychology, Vol. 7, pp.141-164.
12. Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T. and Boyle, J.M. (2012) 'A systematic literature review of empirical evidence on computer games and serious games', Computers and Education, Vol. 59, No. 2, pp.661-686, doi:10.1016/j.compedu.2012.03.004.
13. Connolly, T.M., Stansfield, M.H. and Hainey, T. (2009) 'Towards the development of a gamesbased learning evaluation framework', Connolly, T.M., Stansfield, M.H. and Boyle, E. (Eds.): Games-based Learning Advancement for Multisensory Human Computer Interfaces: Techniques and Effective Practices, Idea-Group Publishing, Hershey. ISBN: 978-1-60566-360-9.
14. Csikszentmihali, M. (1990) Flow: The Psychology of Optimal Experience, Harper and Row, New York.
15. Dankbaar, M.E.W., Bakhuys Roozeboom, M., Oprins, E.A.P.B., Rutten, F., van Saase, J.J.L.C.M., van Merrienboer, J.J.G., Schuit, S.C.E. (2014a) 'Gaming as a training tool to train cognitive skills in emergency care: how effective is it?', in Schouten, B. et al. (Eds.): Games for Health, Proceedings of the 4th Conference, Springer, New York, pp.13-15.
16. Dankbaar, M.E.W., Stegers-Jager, K.M., Baarveld, F., van Merrienboer, J.J.G., Norman, G.R., Rutten, F.L., van Saase, J.L.C.M. Schuit, S.C.E. (2014b), 'Assessing the assessment in emergency care training', PLoS One, Vol. 9, No. 12, p.e114663.
17. Dankbaar, M.E.W. (2015) 'Serious games and blended learning; effects on performance and motivation in medical education', Thesis, Erasmus University Rotterdam, the Netherlands (ch. 4), pp.61-78.
18. Deci, E.L., Eghrari, H., Patrick, B.C. and Leone, D. (1994) 'Facilitating internalization: the selfdetermination theory perspective'. Journal of Personality, Vol. 62, pp.119-142.
19. Egenfeldt-Nielsen, S. (2006), 'Overview of research on the educational use of video games', Digital Kompetanse, Vol. 1, No. 3, pp.184-213.
20. Garris, R., Ahlers, R. and Driskell, J.E. (2002) 'Games, motivation and learning: a research and practice model', Simulation and Gaming: An Interdisciplinary Journal, Vol. 33, pp.441-467.
21. Gee, J.P. (2003) What Videogames Have To Teach Us About Learning And Literacy, Palgrave Macmillan, New York.
22. Gee, J.P. (2009) 'Deep learning properties of good digital games. How far can they go?', In Ritterfeld, U., Cody, M. and Vorderer, P. (Eds.): Serious Games: Mechanisms and Effects, Routledge, New York/London, pp.68-82.
23. Graesser, A.C., Chipman, P., Leeming, F. and Biedenbach, S. (2009) 'Deep learning and emotion in serious games', in Ritterfeld, U., Cody, M. and Vorderer, P. (eds.): Serious Games: Mechanisms and Effects, Routledge, Taylor and Francis, New York and London, pp.81-100.
24. Harteveld, C. (2012), Making Sense of Virtual Risks: A Quasi-Experimental Investigation into Game-Based Training, IOS Press, Amsterdam.
25. Hockey, G.R. (1997), 'Compensatory control in the regulation of human performance under stress and high workload: a cognitive-energetical framework', Biological Psychology, Vol. 45, pp.73-93.
26. Holton, E.F. (1996), 'The flawed four-level evaluation model', Human Resource Development Quarterly, Vol. 7, pp.5-21.
27. Kirkpatrick, D.L. (1976) 'Evaluation of training', in Craig, R.L. (Ed.): Training and Development Handbook: A Guide to Human Resource Development, McGraw Hill, New York, pp.301-319.
28. Kirkpatrick, D.I. (1998) Evaluating Training Programs: The Four Levels, 2nd ed., Berrett-Koehler, San Francisco.
29. Korteling, J.E., Helsdingen, A.S. and Theunissen, N.C.M. (2012) 'Serious games @ work: learning job-related competences using serious gaming', in Bakker, A. and Derks, D. (Eds.): The Psychology of Digital Media at Work, Psychology Press LTD, Taylor and Francis Group, pp.123-144
30. Korteling, J.E., Oprins, E.A.P.B. and Venrooij, W. (2014) Evaluatie van leerinterventies en teamfunctioneren in dynamische teams [evaluation of learning interventions and team functioning in dynamic teams] (TNO report R10243), TNO, Soesterberg.
31. Kraiger, K. (2002), 'Decision-based evaluation', in Kraiger, K. (ed): Creating, Implementing and Managing Effective Training and Development, Jossey-Bass, San Francisco CA, pp.331-375.
32. Kraiger, K., Ford, J.K. and Salas, E. (1993) 'Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation', Journal of Applied Psychology, Vol. 78, No. 2, pp.311-328.
33. Leemkuil, H., de Jong, T. and Ootes, S. (2000) Review of Educational Use of Games and Simulations, (IST-1999-13078 Deliverable D1), University of Twente, Enschede.
34. Malone, T.W. (1981) 'Towards a theory of intrinsically motivating instruction', Cognitive Science, Vol. 4, pp.333-369.
35. Mayer, I. (2012) 'Towards a comprehensive methodology for the research and evaluation of serious games', Procedia Computer Science, Vol. 15, pp.233-247.
36. McMahan, A. (2003) 'Immersion, engagement, and presence: a method for analyzing 3-d video games', in Wolf, M.J.P. and Perron, B. (Eds.): The Video Game, Routledge, Taylor and Francis Group, New York, pp.77-78.
37. Oprins, E. and Korteling, H. (2014) 'Transfer of gaming: effectiveness of a cashier trainer', in Cai, Y.Y. and Goei, S.L. (Eds.): Serious Games, Simulation and Their Applications, Springer, Science and Business Media, Singapore. pp.227-253
38. Pavlas, D. (2010) A Model of Flow and Play in Game-Based Learning: The Impact of Game Characteristics, Player Traits, and Player States, Unpublished PhD thesis, University of Central Florida.
39. Pintrich, P.R. and de Groot, W.V. (1990) 'Motivational and self-regulated learning components of classroom academic performance, Journal of Educational Psychology, Vol. 82, No. 1, pp.33-40.
40. Reeve, J., Jang, H., Carrell, D., Jeon, S. and Barch, J. (2004) 'Enhancing students' engagement by increasing teachers' autonomy support', Motivation and Emotion, Vol. 28, No. 2, pp.147-169.
41. Richardson, M., Abraham, C. and Bond, R. (2012) 'Psychological correlates of university students' academic performance: a systematic review and meta-analysis', Psychological Bulletin, Vol. 138, No. 2, pp.353-387.
42. Roe, R.A. (2005) 'The design of selection systems: contexts, principles, issues', in Evers, A., Smit, O. and Anderson, N. (eds.): Handbook of Personnel Selection, Blackwell, Oxford, pp.73-97.
43. Ryan, R.M. and Deci, E.L. (2000), 'Intrinsic and extrinsic motivations: classic definitions and new directions', Contemporary Educational Psychology, Vol. 25, No. 1, pp.54-67.
44. Salas, E., Milham, L.M. and Bowers, C.A. (2003), 'Training evaluation in the military: misconceptions, opportunities, and challenges', Military Psychology, Vol. 15, pp.3-16.
45. Shaffer. (2006) How Computer Games Help People Learn, Palgrave Macmillan, New York.
46. Sitzmann, T. (2011) 'A meta-analytic examination of the instructional effectiveness of computerbased simulation games', Personnel Psychology, Vol. 64, pp.489-528.
47. Squire, K. D. (2011) Video Games and Learning: Teaching and Participatory Culture in the Digital Age, Teachers College Press, New York.
48. Stubbé, H.M. and Theunissen, N.C.M. (2008) 'Self-directed learning in a ubiquitous learning environment: a meta-review', Proceedings of Special Track on Technology Support for Self-Organised Learners, Vol. 2008, pp.5-28.
49. Sweetser, P. and Wyeth, P. (2005), 'Game flow: a model for evaluating player enjoyment in games', Computers in Entertainment, Vol. 3, No. 3, pp.1-24.
50. Tannenbaum, S., Cannon-Bowers, J., Salas, E. and Mathieu, J. (1993) Factors That Influence Training Effectiveness: A Conceptual Model And Longitudinal Analysis (Tech. Rep. No. 93-011), Naval Training Systems Center, Human Systems Integration Division, Orlando.
51. Tobias, S. and Fletcher, J.D. (2007) 'What research has to say about designing computer games for learning', Educational Technology, Vol. 47, pp.20-29.
52. Trooster, W., Goei, S.L., Ticheloven, A. Oprins, E., Visschedijk, G., Corbalan, G., and Schaik, M. van (2014) 'The effectiveness of LINGO online, a serious game for English pronunciation'. Report Windesheim University of Applied Sciences, Zwolle.
53. Vogel, J., Vogel, D.S., Cannon-Bowers, J., Bowers, C.A., Muse, K. and Wright, M. (2006) 'Computer gaming and interactive simulations for learning: a meta-analysis', Journal of Educational Computing Research, Vol. 34, pp.229-243.
54. Wang, H., Shen, C. and Rittefeld, U. (2009) 'Enjoyment of digital games what makes them 'seriously' fun?', In Ritterfeld, U., Cody, M. and Vorderer, P. (Eds.): Serious Games: Mechanisms and Effects, Routledge, New York/London, pp.25-47.
55. Zijlstra, F.R.H. (1993) Efficiency in Work Behaviour: A Design Approach for Modern Tools, Unpublished PhD Thesis, University of Delft.