The research and evaluation of serious games: Toward a comprehensive methodology

British Journal of Educational Technology

Volumn 45, Issue 3, 2014, Pages 502-527

  Mayer, Igor ^a,b   Bekebrede, Geertje ^a,c   Harteveld, Casper ^d   Warmelink, Harald ^a   Zhou, Qiqi ^a   Van Ruijven, Theo ^a   Lo, Julia ^a   Kortmann, Rens ^a   Wenzler, Ivo ^a,e  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^c Tygron Serious Gaming ^*  (Netherlands)

^d Northeastern University   (Netherlands)

^e ACCENTURE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DESIGN;

ADVANCED LEARNING; COMPUTER BASED SIMULATION; DESIGN PRINCIPLES; RESEARCH AND EVALUATION; RESEARCH QUESTIONS; SCIENTIFIC EVALUATIONS; SCIENTIFIC METHODOLOGY; WORK ORGANIZATION;

RESEARCH;

EID: 84898831621     PISSN: 00071013     EISSN: 14678535     Source Type: Journal    
DOI: 10.1111/bjet.12067     Document Type: Article

References (150)

1. Abt, C., (1970). Serious games. New York: Viking Press.
2. Adams, S. A., (2010). Use of "serious health games" in health care: a review. Studies in Health Technology and Informatics, 157, 160-166. Retrieved May 3, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/20543383
3. Admiraal, W., Huizenga, J., Akkerman, S., &, Dam, G. T., (2011). The concept of flow in collaborative game-based learning. Computers in Human Behavior, 27, 3, 1185-1194. doi: 10.1016/j.chb.2010.12.013.
4. Ainley, M., &, Armatas, C., (2006). Motivational perspectives on students ' responses to learning in virtual learning environments. In, J. Weiss, J. Nolan, J. Hunsinger, &, P. Trifonas, (Eds), The international handbook of virtual learning environments (pp. 365-394). Dordrecht, the Netherlands: Springer. doi: 10.1007/978-1-4020-3803-7-15.
5. Ashton, M. C., &, Lee, K., (2009). An investigation of personality types within the HEXACO personality framework. Journal of Individual Differences, 30, 4, 181-187. doi: 10.1027/1614-0001.30.4.181.
6. Baba, D. M., (1993). Determinants of video game performance. (J. L. Starkes & F. Allard, Eds.) Advances in Psychology, 102, 57-74. Retrieved May 3, 2013, from http://ezproxy.library.unlv.edu/login?url=http://search. ebscohost.com/login.aspx?direct=true%26db=psyh%26AN=1993-98813-004%26site= ehost-live
7. Babbie, E., (2007). The ethics and politics of social research. In, E. Babbie, (Ed.), The practice of social research (pp. 60-83). London: Thomson Wadsworth.
8. Barlett, C. P., Anderson, C. A., &, Swing, E. L., (2008). Video game effects-confirmed, suspected, and speculative: a review of the evidence. Simulation & Gaming, 40, 3, 377-403. doi: 10.1177/1046878108327539.
9. Barnaud, C., Promburom, T., Trebuil, G., &, Bousquet, F., (2007). An evolving simulation/gaming process to facilitate adaptive watershed management in northern mountainous Thailand. Simulation & Gaming, 38, 3, 398-420. doi: 10.1177/1046878107300670.
10. Bekebrede, G., (2010). August 24). Experiencing complexity: a gaming approach for understanding infrastructure systems. TU Delft. Retrieved May 3, 2013, from http://www.narcis.nl/publication/RecordID/oai:tudelft.nl:uuid: dae75f36-4fb6-4a53-8711-8aab42378878
11. Bekebrede, G., Warmelink, H. J. G., &, Mayer, I. S., (2011). Reviewing the need for gaming in education to accommodate the net generation. Computers & Education, 57, 2, 1521-1529. doi: 10.1016/j.compedu.2011.02.010.
12. BGin-Benelux Game Initiative (n.d.). Retrieved December 20, 2012, from http://www.bgin.org/
13. Björk, S., &, Holopainen, J., (2005). Patterns in game design. p. 423. Boston: Charles River Media.
14. Bloom, B. S., Hastings, T. J., &, Madaus, G. F., (1972). Handbook on formative and summative evaluation of student learning. Studies in Art Education, 14, 68-72.
15. Blumberg, F. C., (2000). The effects of children's goals for learning on video game performance. Journal of Applied Developmental Psychology, 21, 6, 641-653. doi: 10.1016/S0193-3973(00)00058-7.
16. Blunt, R. D., (2006). A causal-comparative exploration of the relationship between game-based learning and academic achievement: teaching management with video games. Minneapolis, MN: Walden University. Retrieved May 3, 2013, from http://www.rickblunt.com/phd/blunt-richard-dissertation-final.pdf
17. Boyle, E. A., &, Connolly, T. M., (2008). Games for learning: does gender make a difference? 2nd European Conference on Games Based Learning, 69-76524.
18. Boyle, E. A., Connolly, T. M., &, Hainey, T., (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2, 2, 69-74. doi: 10.1016/j.entcom.2010.12.002.
19. Boyle, E. A., Connolly, T. M., Hainey, T., &, Boyle, J. M., (2012). Engagement in digital entertainment games: a systematic review. Computers in Human Behaviour, 28, 3, 771-780.
20. Bremson, J., (2012). Using gaming simulation to explore long range fuel and vehicle transitions, University of California, California. Retrieved May 3, 2013, from http://steps.ucdavis.edu/its-publications/dissertations/Bremson-Joel. pdf
21. Brockmyer, J. H., Fox, C. M., Curtiss, K., McBroom, E., Burkhart, K. M., &, Pidruzny, J. N., (2009). The development of the Game Engagement Questionnaire: a measure of engagement in video game-playing. Journal of Experimental Social Psychology, 45, 4, 624-634. doi: 10.1016/j.jesp.2009.02.016.
22. Brockner, J., &, Higgins, E. T., (2001). Regulatory focus theory: implications for the study of emotions at work. Organizational Behavior and Human Decision Processes, 86, 1, 35-66. doi: 10.1006/obhd.2001.2972. (Pubitemid 33098543)
23. Brown, D. J., Ley, J., Evett, L., &, Standen, P., (2011). Can participating in games based learning improve mathematic skills in students with intellectual disabilities? 2011 IEEE 1st International Conference on Serious Games and Applications for Health SeGAH (pp. 1-9). IEEE. doi: 10.1109/SeGAH.2011.6165461.
24. Brown, M., Hall, L. R., Holtzer, R., Brown, S., &, Brown, N., (1997). Gender and video game performance. Sex Roles, 36, 11, 793-812. doi: 10.1023/A:1025631307585.
25. Campbell, D. T., &, Stanley, J. C., (1963). Experimental and quasi-experimental designs for research on teaching. In, N. L. Gage, (Ed.), Handbook of research on teaching (pp. 171-246). Chicago: Rand McNally.
26. Carmeli, A., Brueller, D., &, Dutton, J. E., (2009). Learning behaviours in the workplace?: The role of high-quality interpersonal relationships and psychological safety. Systems Research and Behavioral Science, 26, November 2008, 81-98. doi: 10.1002/sres.
27. Castronova, E., (2005). Synthetic worlds: the business and culture of online games. Chicago: University of Chicago Press.
28. Chandler, D., &, Torbert, B., (2003). Condescending ethics and action research. Action Research, 1, 2, 37-47. Retrieved May 3, 2013, from http://arj.sagepub.com/cgi/content/abstract/4/1/37
29. Chertoff, D. B., Goldiez, B., &, LaViola, J. J., (2010). Virtual Experience Test: a virtual environment evaluation questionnaire. 2010 IEEE Virtual Reality Conference VR (pp. 103-110). IEEE. doi: 10.1109/VR.2010.5444804.
30. Chong, D., &, Druckman, J. N., (2007). Framing theory. Annual Review of Political Science, 10, 1, 103-126. doi: 10.1146/annurev.polisci.10.072805. 103054.
31. Climategame (n.d.). (Serious urban planning and water management game). The Hague: Tygron. Retrieved May 3, 2013, from http://climategame.nl/
32. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., &, Boyle, J. M., (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59, 2, 661-686. doi: 10.1016/j.compedu.2012.03.004.
33. Connolly, T. M., Stansfield, M., &, Hainey, T., (2009). Towards the development of a games-based learning evaluation framework. Games-based learning advancements for multisensory human compuiter interfaces: techniques and effective pratices.
34. Cook, T. D., &, Campbell, D. T., (1979). Quasi-experimentation: design and analysis issues for field settings. Boston: Houghlin Mifflin Company. Boston, MA: Houghton Mifflin. (p. 405). Retrieved May 3, 2013, from http://www.amazon.ca/exec/obidos/redirect?tag=citeulike09-20&path=ASIN/ 0395307902
35. Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., etal (2010). Predicting protein structures with a multiplayer online game. Nature, 466, 7307, 756-760. doi:10.1038/nature09304. Retrieved May 3, 2013, from http://www.nature.com/nature/journal/v466/n7307/abs/nature09304. html#supplementary-information
36. Coulthard, G. J., (2009). A review of the educational use and learning effectiveness of simulations and games. Business. West Lafayette, IN: Purdue University. Retrieved May 3, 2013, from http://www.coulthard.com/library/paper- simulation.html
37. Creswell, J. W., (2002). Research design: qualitative, quantitative, and mixed methods approaches (2nd edition). Organizational research methods. Vol. 6 (p. 246). Thousand Oaks, CA: Sage Publications, Inc. Retrieved May 3, 2013, from http://www.amazon.com/Research-Design-Qualitative-Quantitative-Approaches/dp/ 0761924426
38. Csikszentmihalyi, M., (1991). Flow: the psychology of optimal experience. New York: Harper Perennial.
39. De Freitas, S., &, Oliver, M., (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers & Education, 46, 3, 249-264. Retrieved May 3, 2013, from http://www.sciencedirect.com/science/article/B6VCJ-4HVDYJY-2/2/ 821ec9dbad4ba57a1f779f62ba2b0e57 (Pubitemid 43132874)
40. De Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., &, Poulovassilis, A., (2010). Learning as immersive experiences: using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology, 41, 1, 69-85. doi: 10.1111/j.1467-8535.2009.01024.x.
41. De Gloria, A., Bellotti, F., &, Berta, R., (2012). Building a comprehensive R&D community on serious games. Procedia Computer Science, 15, 1-3. doi: 10.1016/j.procs.2012.10.051.
42. De Vaus, D., (2001). Research design in social research. British Educational Research Journal, 28, 296. Sage. Retrieved May 3, 2013, from http://www.amazon.com/Research-Design-Social-David-Vaus/dp/0761953477
43. DiGRA-Digital Games Research Association (n.d.). Retrieved December 20, 2012, from http://www.digra.org/
44. Duke, R. D., (1974). Gaming: the future's language (1st ed., p. 223). New York: Sage Publications.
45. Duke, R. D., (1981). Slogan. (Simulation game for team training), Simulation Experience. Retrieved May 3, 2013, from http://www.simxp.com/en/ simulations-games/slogan/
46. Edmondson, A. C., (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44, 2, 350. doi: 10.2307/2666999.
47. Egenfeldt-Nielsen, S., (2005). Beyond edutainment: exploring the educational potential of computer games. department of digital aesthetics & communication. Copenhagen: IT-University of Copenhagen. Retrieved May 3, 2013, from http://www.seriousgames.dk/downloads/egenfeldt.pdf
48. Egenfeldt-Nielsen, S., (2006). Overview of research on the educational use of video games. Digital Kompetanse, 1, 184-213. Retrieved May 3, 2013, from http://www.idunn.no/ts/dk/2006/03/overview-of-research-on-the-educationaluseof- video-games?languageId=2
49. Enochsson, L., Isaksson, B., Tour, R., Kjellin, A., Hedman, L., Wredmark, T., etal (2004). Visuospatial skills and computer game experience influence the performance of virtual endoscopy. Journal of Gastrointestinal Surgery Official Journal of the Society for Surgery of the Alimentary Tract, 8, 7, 876-882; discussion 882. Retrieved May 3, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/ 15531242
50. Erfani, M., El-Nasr, M., Milam, D., Aghabeigi, B., Lameman, B., Riecke, B. E., etal (2010). The effect of age, gender, and previous gaming experience on game play performance. In, P. Forbrig, F. Paternõ, &, A. Mark Pejtersen, (Eds), Human-computer interaction Vol. 332 (pp. 293-296). Boston: Springer. doi: 10.1007/978-3-642-15231-3-33.
51. Ermi, L., &, Mäyrä, F., (2003). Power and control of games: children as the actors of game cultures. Digital Games Research Conference (Digra) (pp. 234-244). University of Utrecht. Retrieved May 3, 2013, from http://www.uta.fi/∼tlilma/Ermi-Mayra-Power-and-Control-of-Games.pdf
52. Ferris, G. R., (2005). Development and validation of the political skill inventory. Journal of Management, 31, 1, 126-152. doi: 10.1177/0149206304271386. (Pubitemid 41094196)
53. GALA-Network of Excellence for Serious Games (n.d.). Retrieved May 3, 2013, from http://www.galanoe.eu/
54. Garris, R., Ahlers, R., &, Driskell, J. E., (2002). Games, motivation, and learning: a research and practice model. Simulation & Gaming, 33, 4, 441-467. doi: 10.1177/1046878102238607.
55. Geurts, J. L. A., Duke, R. D., &, Vermeulen, P. A. M., (2007). Policy gaming for strategy and change. Long Range Planning, 40, 6, 535-558. doi: 10.1016/j.lrp.2007.07.004. (Pubitemid 350148972)
56. Girard, C., Ecalle, J., &, Magnan, A., (2012). Serious games as new educational tools: how effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 19, 3, 207-219. doi: 10.1111/j.1365-2729. 2012.00489.x.
57. Good, B. M., &, Su, A. I., (2011). Games with a scientific purpose. Genome Biology, 12, 12, 135. doi: 10.1186/gb-2011-12-12-135.
58. Gosen, J., &, Washbush, J., (2004). A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35, 2, 270-293. doi: 10.1177/1046878104263544.
59. Greenblat, C. S., (1973). Teaching with simulation games: a review of claims and evidence. (C. Greenblat & R. D. Duke, Eds.) Teaching Sociology, 1, 1, 62-83.
60. Guay, F., Vallerand, R. J., &, Blanchard, C., (2000). On the assessment of situational intrinsic and extrinsic motivation: the Situational Motivation Scale (SIMS). Motivation and Emotion, 24, 3, 175-213.
61. Hainey, T., (2010). Using games-based learning to teach requirements collection and analysis at tertiary education level. Edinburgh: University of the West of Scotland. Retrieved May 3, 2013, from http://scholar.google.com/ scholar?hl=en&btnG=Search&q=intitle:Using+Games- Based+Learning+to+Teach+Requirements+Collection+and+Analysis+at+Tertiary+Educat ion+Level#1
62. Hainey, T., &, Connolly, T. M., (2010). Evaluating games-based learning. International Journal of Virtual and Personal Learning Environments, 1, 1, 57-71. doi: 10.4018/jvple.2010091705.
63. Harper, J. D., Kaiser, S., Ebrahimi, K., Lamberton, G. R., Hadley, H. R., Ruckle, H. C., etal (2007). Prior video game exposure does not enhance robotic surgical performance. Journal of Endourology / Endourological Society, 21, 10, 1207-1210. doi: 10.1089/end.2007.9905. (Pubitemid 47614369)
64. Harteveld, C., (2011). Triadic game design. London: Springer. Retrieved May 3, 2013, from http://books.google.nl/books?id=sIcQYKcoH7IC
65. Harteveld, C., (2012). Making sense of virtual risks; a quasi experimental investigation into game-based training. (p. 435). Delft, The Netherlands: IOS Press. doi: 10.4233/uuid:c3d38ad9-88cb-403c-b69e-ba22b005a26f.
66. Hays, R. T., (2005). The effectiveness of instructional games: a literature review and discussion. Orlando: Naval War Center: Training Systems Division.
67. Hofstede, G., (1986). Cultural differences in teaching and learning. (G. Hofstede, Ed.) International Journal of Intercultural Relations, 10, 3, 301-320. doi: 10.1016/0147-1767(86)90015-5.
68. Hofstede, G. J., De Caluwe, L., &, Peters, V. A. M., (2010). Why simulation games work-in search of the active substance: a synthesis. Simulation & Gaming, 41, 6, 824-843. doi: 10.1177/1046878110375596.
69. Holsbrink-Engels, G. A., (1998). Computer-based role playing for interpersonal skills training. Enschede: Universiteit Twente.
70. Huang, W.-H., (2011). Evaluating learners' motivational and cognitive processing in an online game-based learning environment. Computers in Human Behaviour, 27, 2, 694-704. doi: 10.1016/j.chb.2010.07.021.
71. ISAGA (n.d.). Retrieved May 3, 2013, from http://www.isaga.com/
72. Jackson, C. L., Colquitt, J., Wesson, M. J., &, Zapata-Phelan, C. P., (2006). Psychological collectivism: a measurement validation and linkage to group member performance. The Journal of Applied Psychology, 91, 4, 884-899. doi: 10.1037/0021-9010.91.4.884. (Pubitemid 44128211)
73. Jenson, J., &, De Castell, S., (2010). Gender, simulation, and gaming: research review and redirections. Simulation & Gaming, 41, 1, 51-71. doi: 10.1177/1046878109353473.
74. Kato, P. M., Cole, S. W., Bradlyn, A. S., &, Pollock, B. H., (2008). A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics, 122, 2, e305-e317. doi: 10.1542/peds.2007-3134.
75. Ke, F., (2009). A qualitative meta-analysis of computer games as learning tools (chapter 1) (pp. 1-33). IGI.
76. Kenny, R. F., &, Gunter, G., (2011). Factors affecting adoption of video games in the classroom. Journal of Interactive Learning Research, 22, 2, 259-276. Retrieved May 3, 2013, from http://www.editlib.org/p/32258
77. Kenny, R. F., &, McDaniel, R., (2011). The role teachers' expectations and value assessments of video games play in their adopting and integrating them into their classrooms. British Journal of Educational Technology, 42, 2, 197-213. doi: 10.1111/j.1467-8535.2009.01007.x.
78. Kickmeier-Rust, M. D., Steiner, C. M., &, Albert, D., (2009). Non-invasive assessment and adaptive interventions in learning games. Intelligent Networking and Collaborative Systems 2009 INCOS 09 International Conference on (pp. 301-305). IEEE. doi: 10.1109/INCOS.2009.30.
79. Kinzie, M. B., &, Joseph, D. R. D., (2008). Gender differences in game activity preferences of middle school children: implications for educational game design. Educational Technology Research & Development, 56, 5-6, 643-663. doi: 10.1007/s11423-007-9076-z.
80. Kirkpatrick, D. L., (1994). Evaluating training programs: the four levels. San Francisco: Berret-Koehler.
81. Kirkpatrick, D. L., (2006). Seven keys to unlock the four levels of evaluation. Performance Improvement, 45, 7, 5-8.
82. Knight, J. F., Carley, S., Tregunna, B., Jarvis, S., Smithies, R., De Freitas, S., etal (2010). Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation, 81, 9, 1175-1179. Retrieved May 3, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/20732609
83. Kortmann, R., Bekebrede, G., Van Daalen, C. E., Harteveld, C., Mayer, I. S., &, Van Dierendonck, D., (2012). Veerkracht-a game for servant-leadership development. Proceedings of the 43rd annual conference of the International Simulation and Gaming Association.
84. Kriz, W. C., &, Hense, J. U., (2004). Evaluation of the EU-project "Simgame" in business education. In, W. C. Kriz, &, T. Eberle, (Eds), Bridging the gap: transforming knowledge into action through gaming and simulation (Isaga 2004 Conference Proceedings) (pp. 352-363). Munchen: Sagsaga.
85. Kriz, W. C., &, Hense, J. U., (2006). Theory-oriented evaluation for the design of and research in gaming and simulation. Simulation & Gaming, 37, 2, 268-283. doi: 10.1177/1046878106287950. (Pubitemid 43676705)
86. Kuit, M., (2002). Strategic behavior and regulatory styles in the Netherlands Energy Industry. Retrieved May 3, 2013, from http://www.narcis.nl/ publication/RecordID/oai:tudelft.nl:uuid:f940a515-f967-4cf3-9a96-07cb69a61ae5
87. Lee, J., (1999). Effectiveness of computer-based instructional simulation: a meta analysis. International Journal of Instructional Media, 26, 1, 71-85. Retrieved May 3, 2013, from http://www.questia.com/googleScholar.qst? docId=5001238108
88. Lee, K., &, Ashton, M. C., (2004). Psychometric properties of the HEXACO personality inventory. Multivariate Behavioral Research, 39, 2, 329-358. doi: 10.1207/s15327906mbr3902-8. (Pubitemid 39245170)
89. Leemkuil, H., (2006). Is it all in the game? Learner support in an educational knowledge management simulation game. Enschede, The Netherlands: University of Twente. Retrieved May 3, 2013, from http://users.gw.utwente.nl/ leemkuil/PhDThesisLeemkuil2006.pdf
90. Lepper, M., &, Malone, T. W., (1987). Intrinsic motivation and instructional effectiveness in computer-based education. In, R. E. Snow, &, M. J. Farr, (Eds), Aptitude, learning, and instruction. III. Cognitive and affective process analysis (pp. 255-286). Hillsdale, NJ: Lawrence Erlbaum Associates Publishers.
91. Leyton-Brown, K., &, Shoham, Y., (2008). Essentials of game theory. In, R. J. Brachman, &, T. Dietterich, (Eds), Political science Vol. 2 (p. 104). San Francisco, CA: Morgan & Claypool Publishers. doi: 10.2200/S00108ED1V01Y200802AIM003.
92. Ma, Y., Williams, D., Prejean, L., &, Richard, C., (2007). A research agenda for developing and implementing educational computer games: colloquium. British Journal of Educational Technology, 38, 3, 513-518. (Pubitemid 46645588)
93. Mackenzie, N., &, Knipe, S., (2006). Research dilemmas: paradigms, methods and methodology. Issues In Educational Research, 16, 2, 193-205. Retrieved May 3, 2013, from http://www.iier.org.au/iier16/mackenzie.html
94. Malone, T. W., &, Lepper, M., (1987). Making learning fun: a taxonomy of intrinsic motivation for learning. In, R. E. Snow, &, M. J. Farr, (Eds), Aptitude, learning, and instruction: III. Cognitive and affective process analysis (pp. 223-225). Hillsdale, NJ: Lawrence Erlbaum Associates Publishers.
95. Martin, A. J., &, Jackson, S., (2008). Brief approaches to assessing task absorption and enhanced subjective experience: examining "short" and "core" flow in diverse performance domains. Motivation and Emotion, 32, 3, 141-157. doi: 10.1007/s11031-008-9094-0.
96. Mayer, I. S., (2009). The gaming of policy and the politics of gaming: a review. (Y. G. Kin, C. Yiyu, & Y. Gee, Eds.) Simulation & Gaming, 40, 6, 825-862. doi: 10.1177/1046878109346456.
97. Mayer, I. S., Carton, L., De Jong, M., Leijten, M., &, Dammers, E., (2004). Gaming the future of an urban network. Futures, 36, 3, 311-333. doi: 10.1016/S0016-3287(03)00159-9. (Pubitemid 38122856)
98. Mayer, I. S., Warmelink, H. J. G., &, Bekebrede, G., (2013). Learning in a game-based virtual environment: a comparative evaluation in higher education. European Journal of Engineering Education, 38, 1, 85-106. doi: 10.1080/03043797.2012.742872.
99. Mayer, I. S., Zhou, Q., Lo, J., Abspoel, L., Keijser, X., Olsen, E., etal (2012). Integrated, ecosystem-based marine spatial planning?: First results from international simulation-game experiment. Third international engineering systems symposium CESUN 2012, Delft University of Technology, 18-20 June 2012 (pp. 18-20). Delft, The Netherlands: IEEE.
100. Mayer, I. S., Zhou, Q., Lo, J., Abspoel, L., Keijser, X., Olsen, E., etal (in press). Integrated, ecosystem-based marine spatial planning: design and results of a game-based quasi-experiment. Ocean and Coastal Management. doi:10.1016/j.ocecoaman.2013.04.006
101. Mayer, R. E., (2005). Cognitive theory of multimedia learning. In, R. E. Mayer, (Ed.), The Cambridge handbook of multimedia learning (pp. 31-48). Cambridge, MA: Cambridge University Press. Retrieved May 3, 2013, from http://books.google.com/books?hl=en&lr=&id=duWx8fxkkk0C&oi= fnd&pg=PA31&dq=Cognitive+Theory+of+Multimedia+Learning&ots= x62cq2Ag4C&sig=km1JhOxQR2OGZunazZVGY6qDU3Q
102. Mayes, D. K., &, Cotton, J. E., (2001). Measuring engagement in video games: a questionnaire. Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting (pp. 692-696). Retrieved May 3, 2013, from http://www.scopus.com/inward/record.url?eid=2-s2.0-0442310948&partnerID= 40&md5=b56e3e59c17f58d4d21a35d7755b9f6a
103. McGonigal, J. E., (2011). Reality is broken: why games make us better and how they can change the world. Vol. 22 Penguin Books, Ed. New York: The Penguin Press HC. (p. 400). Retrieved May 3, 2013, from http://www.amazon.com/dp/ 1594202850
104. Meyer, B., (2010). Comparative studies in game-based language learning: a discussion of methodology. (P. Escudeiro, Ed.) Proceedings of the 9th European Conference on ELearning Vol 1 (pp. 362-369).
105. Mortagy, Y., &, Boghikian-whitby, S., (2010). A longitudinal comparative study of student perceptions in online education. Interdisciplinary Journal of Elearning and Learning Objects, 6, 23-44. Retrieved May 3, 2013, from http://ijello.org/Volume6/IJELLOv6p023-044Mortagy684.pdf
106. Noy, A., Raban, D. R., &, Ravid, G., (2006). Testing social theories in computer-mediated communication through gaming and simulation. Simulation & Gaming, 37, 2, 174-194. doi: 10.1177/1046878105286184. (Pubitemid 43676700)
107. Olsen, T., Procci, K., &, Bowers, C., (2011). Serious games usability testing: how to ensure proper usability, playability, and effectiveness. (A. Marcus, Ed.) Design User Experience and Usability Theory Methods Tools and Practice Proceedings First International Conference DUXU 2011 Held as Part of HCI International 2011, 6770, 625-634. doi: 10.1007/978-3-642-21708-1.
108. Oslin, J. L., Mitchell, S. A., &, Griffin, L. L., (1998). The Game Performance Assessment Instrument (GPAI): some concerns and solutions for further development. Journal of Teaching in Physical Education, 17, 2, 220-240. Retrieved May 3, 2013, from http://www.cababstractsplus.org/abstracts/Abstract. aspx?AcNo=19981804818
109. Paas, F., Tuovinen, J. E., Merriënboer, J. J. G., &, Aubteen Darabi, A., (2005). A motivational perspective on the relation between mental effort and performance: optimizing learner involvement in instruction. Educational Technology Research and Development, 53, 3, 25-34. doi: 10.1007/BF02504795. (Pubitemid 41137731)
110. Papastergiou, M., (2009). Digital game-based learning in high school computer science education: impact on educational effectiveness and student motivation. Computers & Education, 52, 1, 1-12. doi: 10.1016/j.compedu.2008. 06.004.
111. Pfister, R., (2011). Gender effects in gaming research: a case for regression residuals? Cyberpsychology Behavior and Social Networking, 14, 10, 603-607. doi: 10.1089/cyber.2010.0547.
112. Preschl, B., Wagner, B., Forstmeier, S., &, Maercker, A., (2011). E-health interventions for depression, anxiety disorder, dementia, and other disorders in older adults: a review. Cyber Therapy and Rehabilitation, 4, 3, 371-385. Retrieved May 3, 2013, from http://ovidsp.ovid.com/ovidweb.cgi?T= JS&PAGE=reference&D=psyc7&NEWS=N&AN=2011-25383-006
113. Randel, J. M., Morris, B. A., Wetzelf, C. D., &, Whitehill, B. V., (1992). The effectiveness of games for educational purposes: a review of the research. Simulation & Gaming, 25, 261-276. Retrieved May 3, 2013, from http://www.cofc.edu/∼seay/cb/findsim.html
114. Raphael, C., Bachen, C., Lynn, K. M., Baldwin-Philippi, J., &, McKee, K. A., (2009). Games for civic learning: a conceptual framework and agenda for research and design. Games and Culture, 5, 2, 199-235. doi: 10.1177/ 1555412009354728.
115. Reichlin, L., Mani, N., McArthur, K., Harris, A. M., Rajan, N., &, Dacso, C. C., (2011). Assessing the acceptability and usability of an interactive serious game in aiding treatment decisions for patients with localized prostate cancer. Journal of Medical Internet Research, 13, 1, e4. doi: 10.2196/jmir.1519.
116. Salomon, M., &, Soudoplatoff, S., (2010). Why virtual-world economies matter. Journal of Virtual Worlds Research, 2, 4, 14. Retrieved May 3, 2013, from http://journals.tdl.org/jvwr/article/viewArticle/859
117. Sanchez, A., Cannon-bowers, J., &, Bowers, C., (2010). Establishing a science of game based learning. In, J. A. Cannon-Bowers, &, C. Bowers, (Eds.), Serious game design and development: technologies for training and learning (pp. 290-304). Hershey, PA: IGI Global. doi:10.4018/978-1-61520-739-8. ch016.
118. Scharpf, F. W., (1997). Games real actors play?: actor-centered institutionalism in policy research. In, P. Sabatier, (Ed.), Theoretical lenses on public policy (p. 318). Boulder, CO: Westview Press. Retrieved May 3, 2013, from http://opac.rero.ch/get-bib-record.cgi?db=ne&rero-id=R236641760
119. Schneider, D. K., (2005). Research design for educational technologists. Geneva: AACE. [unpublished working document]. Retrieved May 3, 2013, from http://www.editlib.org/index.cfm?fuseaction=Reader.ViewAbstract&paper-id= 28664
120. Schuurink, E., Houtkamp, J., &, Toet, A., (2008). Engagement and EMG in serious gaming: experimenting with sound and dynamics in the Levee Patroller training game. In, P. Markopoulos, B. de Ruyter, W. IJsselsteijn, &, D. Rowland, (Eds.), Fun and games. Second International Conference, Eindhoven, The Netherlands, October 20-21, 2008. Proceedings (pp. 139-149). Berlin, Heidelberg: Springer Verlag. doi:10.1007/978-3-540-88322-7-14.
121. SG Academy (n.d.). Retrieved December 20, 2012, from http://academy. seriousgamessociety.org/
122. SGA-Serious Games Association (n.d.). Retrieved May 3, 2013, from http://www.seriousgamesassociation.com/
123. Shaffer, D. W., (2006). Epistemic frames for epistemic games. Computers & Education, 46, 3, 223-234. doi: 10.1016/j.compedu.2005.11.003. (Pubitemid 43132872)
124. Sharkworld (n.d.). (Serious game about project management). Rotterdam: Ranj. Retrieved May 3, 2013, from http://www.sharkworld.nl/
125. Shubik, M., (1999). Political economy, oligopoly and experimental games: the selected essays of Martin Shubik volume one. Cheltenham: Elgar.
126. Shute, V. J., (2011). Stealth assessment in computer-based games to support learning. In, S. Tobias, &, J. D. Fletcher, (Eds), Computer games and instruction Vol. 55 (pp. 503-524). Charlotte, NC: Information Age Publishers. Retrieved May 3, 2013, from http://proxygw.wrlc.org/login?url=http:/ /search.ebscohost.com/login.aspx?direct=true%26db=psyh%26AN=2011-11269- 020%26site=eds-live%26scope=site
127. Shute, V. J., Masduki, I., &, Donmez, O., (2010). Conceptual framework for modeling, assessing and supporting competencies within game environments. Technology Instruction Cognition and Learning, 8, 2, 137-161. Retrieved May 3, 2013, from http://myweb.fsu.edu/vshute/pdf/TICL2010.pdf
128. Shute, V. J., Ventura, M., Bauer, M., &, Zapata-Rivera, D., (2009). Melding the power of serious games and embedded assessment to monitor and foster learning?: Flow and grow. (U. Ritterfeld, M. Cody, & P. Vorderer, Eds.) Serious Games Mechanisms and Effects, 1, 1, 1-33. Retrieved May 3, 2013, from http://books.google.com/books?hl=en&lr=&id=GwPf7tbO5mgC&pgis=1
129. SprintCity (n.d.). (Serious game about urban planning). Rotterdam, The Netherlands: Vereniging Deltametropool. Retrieved May 3, 2013, from http://deltametropool.nl/nl/sprintstad-in-utrecht
130. Squire, K., (2002). Cultural framing of computer/video games. Game Studies. The International Journal of Computer Game Research, 2, 1. Retrieved May 3, 2013, from http://www.gamestudies.org/0102/squire/
131. Squire, K., (2004). Replaying history: learning world history through playing civilization III. Bloomington, IN: Indiana University. Retrieved May 3, 2013, from http://website.education.wisc.edu/kdsquire/dissertation.htm
132. Steunenberg, B., Schmidtchen, D., &, Koboldt, C., (1999). Strategic power in the european union: evaluating the distribution of power in policy games. Journal of Theoretical Politics, 11, 3, 339-366. doi: 10.1177/0951692899011003005.
133. Stichting Rechten Online (2007). Cyberdam. Stichting Rechten Online. Retrieved August 10, 2012, from http://www.cyberdam.nl
134. Szturm, T., Betker, A. L., Moussavi, Z., Desai, A., &, Goodman, V., (2011). Effects of an interactive computer game exercise regimen on balance impairment in frail community-dwelling older adults: a randomized controlled trial. Physical Therapy, 91, 10, 1449-1462. doi: 10.2522/ptj.20090205.
135. Tallir, I. B., Lenoir, M., Valcke, M., &, Musch, E., (2007). Do alternative instructional approaches result in different game performance learning outcomes? Authentic assessment in varying game conditions. International Journal of Sport Psychology, 38, 3, 263-282.
136. Trepte, S., &, Reinecke, L., (2011). The pleasures of success: game-related efficacy experiences as a mediator between player performance and game enjoyment. Cyberpsychology Behavior and Social Networking, 14, 9, 555-557. Retrieved May 3, 2013, from http://www.ncbi.nlm.nih.gov/pubmed/21342012
137. TU Delft, Tygron Serious Gaming, & Port of Rotterdam (n.d.). SimPort MV2 serious game. Delft, The Netherlands: Tygron serious games and media. Retrieved May 3, 2013, from http://www.simport.eu/
138. Tykhonov, D., Jonker, C., &, Meijer, S. A., (2008). Agent-based simulation of the trust and tracing game for supply chains and networks. and Social Simulation. Retrieved May 3, 2013, from http://jasss.soc.surrey.ac.uk/11/ 3/1.html
139. Van der Spek, E. D., (2011). (October 26). Experiments in serious game design: a cognitive approach. SIKS Dissertation Series. P. Retrieved May 3, 2013, from http://www.narcis.nl/publication/RecordID/oai:dspace.library.uu.nl: 1874/211480
140. Van der Spek, E. D., Wouters, P., &, Van Oostendorp, H., (2011). Code red: triage or cognition-based design rules enhancing decisionmaking training in a game environment. British Journal of Educational Technology, 42, 3, 441-455. doi: 10.1111/j.1467-8535.2009.01021.x.
141. Vartiainen, P., (2000). Evaluation methods and comparative study. European Evaluation Society EES Conference (pp. 12-14). Retrieved May 3, 2013, from http://cjpe.ca/distribution/20001012-vartiainen-pirkko.pdf
142. Ventum Online (n.d.). (Serious project management game). Delft: TU Delft, Faculteit TBM. Retrieved May 3, 2013, from http://www.ventumonline.nl/
143. Venture Simulations Ltd. (n.d.). Simventure. (serious business game). York, UK: Venture Simulations Ltd. Retrieved May 3, 2013, from http://www.simventure.co.uk/
144. Verdaasdonk, E. G. G., Dankelman, J., Schijven, M. P., Lange, J. F.,
145. Wolfe, J., &, Box, T. M., (1988). Team cohesion effects on business game performance. Simulation & Gaming, 19, 1, 82-98. doi: 10.1177/003755008801900105.
146. Wood, R. T. A., Griffiths, M. D., &, Eatough, V., (2004). Online data collection from video game players: methodological issues. CyberPsychology Behavior, 7, 5, 511-518. doi: 10.1089/cpb.2004.7.511. (Pubitemid 39482470)
147. Wright, J. D., &, Marsden, P. V., (Eds.) (2010). Handbook of survey research (2nd ed.). Bingley, UK: Emerald Group Publishing.
148. Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., etal (2012). Our princess is in another castle: a review of trends in serious gaming for education. Review of Educational Research, 82, 1, 61-89. doi: 10.3102/0034654312436980.
149. Yusoff, A., Crowder, R., &, Gilbert, L., (2010). Validation of serious games attributes using the technology acceptance model. (IEEE, Ed.) 2010 Second International Conference on Games and Virtual Worlds for Serious Applications, 0, 45-51. doi: 10.1109/VS-GAMES.2010.745.
150. Zhou, Q., Mayer, I. S., Bekebrede, G., Warmerdam, J., &, Knepflé, M., (2013). The climate game: connecting water management and spatial planning through simulation gaming? In, J. Edelenbos, N. Bressers, &, P. Scholten, (Eds), Water governance as connective capacity (pp. 109-127). Farnham, Surrey, UK: Ashgate Publishing Ltd.