Measuring expert performance for serious games analytics: From data to insights

Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement

Volumn , Issue , 2015, Pages 101-134

  Loh, Christian Sebastian ^a   Sheng, Yanyan ^a  

^a SOUTHERN ILLINOIS UNIVERSITY   (United States)

Author keywords

Action sequence; Expert novice; Expertise performance index; In situ data; Performance metrics; Similarity measure

Indexed keywords

EID: 84944604420     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-319-05834-4_5     Document Type: Chapter

References (91)

1. Abt, C. C. (1987). Serious games (Reprint). Lanham, MD: University Press of America.
2. Aldrich, C. (2005). Learning by doing: a comprehensive guide to simulations, computer games, and pedagogy in e-learning and other educational experiences. San Francisco: Pfeiffer.
3. Anderson, E. (1957). A semigraphical method for the analysis of complex problems. Proceedings of the National Academy of Sciences of the United States of America, 43(10), 923-927. Retrieved December 12, 2014, from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC528552/
4. Bauer, M. (2002). Using evidence-centered design to align formative and summative assessment. In Proceedings of the Evidence-Centered Design Approach to Creating Diagnostic Assessments (ITS2002) (pp. 87-96).
5. Bellotti, F., Kapralos, B., Lee, K., & Moreno-Ger, P. (2013). User assessment in serious games and technology-enhanced learning. Advances in Human-Computer Interaction, 2013, 2. doi: 10.1155/2013/120791.
6. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013, 11. doi: 10.1155/2013/136864.
7. Ben Zur, H., & Breznitz, S. J. (1981). The effect of time pressure on risky choice behavior. Acta Psychologica, 47(2), 89-104. doi:10.1016/0001-6918(81)90001-9.
8. Bishop, C. M. (2006). Pattern recognition and machine learning. New York, NY: Springer Science+Business Media.
9. Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., & Gratton, G. (2008). The effects of video game playing on attention, memory, and executive control. Acta Psychologica, 129(3), 387-398. doi:10.1016/j.actpsy.2008.09.005.
10. Byun, J. H., & Loh, C. S. (2015). Audial engagement: Effects of game sound on learner engagement in digital game-based learning environments. Computer in Human Behavior, 46, 129-138. doi:10.1016/j.chb.2014.12.052.
11. Canossa, A., & Drachen, A. (2009). Patterns of play: Play-personas in user-centred game development. In Proceedings of Breaking New Ground: Innovation in Games, Play, Practice and Theory Conference. London: DiGRA.
12. Cobb, B. R., Rumi, R., & Salmerón, A. (2007). Bayesian network models with discrete and continuous variables. In P. Lucas, J. A. Gamez, & A. Salmerón (Eds.), Advances in probabilistic graphical models (Studies in fuzziness and soft computing, Vol. 214, pp. 81-102). Berlin, Germany: Springer. doi:10.1007/978-3-540-68996-6_4.
13. Crookall, D. (2010). Serious games, debriefing, and simulation/gaming as a discipline. Simulation & Gaming, 41(6), 898-920. doi:10.1177/1046878110390784.
14. Csikszentmihalyi, M. (1991). Flow: The psychology of optimum experience. New York: Harper Perennial.
15. DeSanctis, G. (1984). Computer graphics as decision aids: Directions for research. Decision Sciences, 15(4), 463-487. doi:10.1111/j.1540-5915.1984.tb01236.x.
16. Diez, F. J., Mira, J., Iturralde, E., & Zubillaga, S. (1997). DIAVAL, a Bayesian expert system for echocardiography. Artificial Intelligence in Medicine, 10(1), 59-73. doi:10.1016/S0933-3657(97)00384-9.
17. Dixit, P. N., & Youngblood, G. M. (2008). Understanding playtest data through visual data mining in interactive 3D environments. In Proceedings of the 12th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia and Serious Games, Louisville (CGAMES 2008). Wolverhampton, England: University of Wolverhampton. Retrieved December 12, 2014, from http://gameintelligencegroup.org/files/2009/09/DixitACMSS2008.pdf
18. Djaouti, D., Alvarez, J., & Jessel, J.-P. (2011). Classifying serious games: The G/P/S model. In P. Felicia (Ed.), Handbook of research on improving learning and motivation through educational games (pp. 118-136). Hershey, PA: IGI Global. doi:10.4018/978-1-60960-495-0.ch006.
19. Drachen, A., & Canossa, A. (2011). Evaluating motion: Spatial user behaviour in virtual environment. International Journal of Arts and Technology, 4(3), 294-314. doi:10.1504/UART.2011.041483.
20. Drachen, A., Canossa, A., & Sørensen, J. R. M. (2013). Gameplay metrics in game user research: Examples from the trenches. In M. S. El-Nasr, A. Drachen, & A. Canossa (Eds.), Game analytics: Maximizing the value of player data (pp. 285-319). London: Springer. doi:10.1007/978-1-4471-4769-5_14.
21. Drachen, A., Thurau, C., Sifa, R., & Bauckhage, C. (2013). A comparison of methods for player clustering via behavioral telemetry. In Proceedings of the 8th International Conference on the Foundations of Digital Games (FDG 2013) (pp. 245-252). Crete, Greece: Society for the Advancement of the Science of Digital Games.
22. Dreyfus, S. E. (2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology and Society, 24(3), 177-181. doi:10.1177/0270467604264992.
23. Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition. Berkeley, CA: University of California. Retrieved December 12, 2014, from http://www.dtic.mil/get-tr-doc/pdf?AD=ADA084551
24. Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (Eds.). (2006). The Cambridge handbook of expertise and expert performance. Cambridge Handbooks in Psychology. New York, NY: Cambridge University Press.
25. Ermi, L., & Mäyrä, F. (2007). Fundamental components of the gameplay experience: Analyzing immersion. In S. de Castell & J. Jenson (Eds.), Worlds in play: International perspectives on digital games research (pp. 37-53). New York: Peter Lang.
26. Estivill-Castro, V. (2002). Why so many clustering algorithms: A position paper. ACM SIGKDD Explorations Newsletter, 4(1), 65-75. doi:10.1145/568574.568575.
27. Fan, X., Miller, B., Park, K.-E., Winward, B. W., Christensen, M., Grotevant, H. D., et al. (2006). An exploratory study about inaccuracy and invalidity in adolescent self-report surveys. Field Methods, 18(3), 223-244. doi:10.1177/152822X06289161.
28. Folkestad, J. E., Robinson, D. H., McKernan, B., Martey, R. M., Rhodes, M. G., & Stormer-Galley S. (2015). Analytics-driven design: Impact and implications of team member psychological perspectives on a serious games (SGs) design framework. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement. New York: Springer.
29. Grimshaw, M., Lindley, C. A., & Nacke, L. (2008). Sound and immersion in the first-person shooter: Mixed measurement of the player’s sonic experience. In Proceedings of the Audio Mostly Conference. Retrieved December 12, 2014, from http://wlv.openrepository.com/wlv/bitstream/2436/35995/2/Grimshaw_CGAMES07.pdf
30. Heckerman, D. (1995). A tutorial on learning with Bayesian networks. Redmond, WA: Microsoft. Retrieved December 12, 2014, from http://research.microsoft.com/pubs/69588/tr-95-06.pdf
31. Hofer, A. (2011). Exploratory comparison of expert and novice pair programmers. In Z. Huzar, R. Koci, B. Meyer, B. Walter, & J. Zendulka (Eds.), Software Engineering Techniques (Vol. 4980, pp. 218-231). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-22386-0_17.
32. Hoskin, R. (2012). The dangers of self-report. Retrieved December 12, 2014, from http://www.sciencebrainwaves.com/uncategorized/the-dangers-of-self-report/
33. IJsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., & Bellotti, F. (2007). Characterising and measuring user experiences in digital games. In Proceedings of the International Conference on Advances in Computer Entertainment Technology (pp. 27-30).
34. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning — with applications in R. New York, NY: Springer New York. doi:10.1007/978-1-4614-7138-7.
35. Jefford, M., Stockler, M. R., & Tattersall, M. H. N. (2003). Outcomes research: What is it and why does it matter? Internal Medicine Journal, 33(3), 110-118. doi:10.1046/j.1445-5994.2003.00302.x.
36. Jonassen, D. H., Hannum, W. H., & Tessmer, M. (1989). Handbook of task analysis procedures. Westport, CT: Praeger Publishers.
37. Joslin, S., Brown, R., & Drennan, P. (2007). The gameplay visualization manifesto. Computers in Entertainment, 5(3), 6. doi: 10.1145/1316511.1316517.
38. Karelaia, N., & Hogarth, R. M. (2008). Skill, luck, overconfidence, and risk taking. Barcelona, Spain: Universitat Pompeu Fabra. doi:10.2139/ssrn.1374235.
39. Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., & Wixon, D. (2008). Tracking Real-time User Experience (TRUE): A comprehensive instrumentation solution for complex systems. In Proceeding of the 26th Annual CHI Conference on Human Factors in Computing Systems (CHI’08) (p. 443). New York: ACM Press. doi:10.1145/1357054.1357126.
40. Kirkley, J., Kirkley, S., & Heneghan, J. (2007). Building bridges between serious game design and instructional design. In B. E. Shelton & D. A. Wiley (Eds.), Educational design & use of computer simulation games (pp. 59-81). Rotterdam, The Netherlands: Sense.
41. Kirriemuir, J., & McFarlane, A. (2003). Use of computer and video games in the classroom. In Proceedings of the Level up Digital Games Research Conference. Utrecht, The Netherlands: Universiteit Utrecht.
42. Lachenbruch, P. A., & Mickey, M. R. (1968). Estimation of error rates in discriminant analysis. Technometrics, 10(1), 1-11. doi: 10.1080/00401706.1968.10490530.
43. Law, B., Atkins, M. S., Kirkpatrick, A. E., & Lomax, A. J. (2004). Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. In Proceedings of the 2004 Symposium on Eye Tracking Research & Applications (ETRA’04) (pp. 41-48). New York, NY: ACM Press. doi: 10.1145/968363.968370.
44. Levin, A. (2010, August 30). Simulator training flaws tied to airline crashes. USA Today. Retrieved December 12, 2014, from http://usatoday30.usatoday.com/travel/flights/2010-08-31-1Acockpits31_ST_N.htm
45. Liu, M., Kang, J., Lee, J., Winzeler, E., & Liu, S. (2015). Examining through visualization what tools learners access as they play a serious game for middle school science. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement. New York: Springer.
46. Loh, C. S. (2006). Designing online games assessment as “Information Trails.”. In D. Gibson, C. Aldrich, & M. Prensky (Eds.), Games and simulation in online learning: Research and development frameworks (pp. 323-348). Hershey, PA: Idea Group. doi:10.4018/978-1-59904-941-0.ch032.
47. Loh, C. S. (2012a). Improving the impact and return of investment of game-based learning. International Journal ofVirtual and Personal Learning Environments, 4(1), 1-15. doi:10.4018/jvple.2013010101.
48. Loh, C. S. (2012b). Information trails: In-process assessment of game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 123-144). New York: Springer. doi:10.1007/978-1-4614-3546-4.
49. Loh, C. S., Anantachai, A., Byun, J. H., & Lenox, J. (2007). Assessing what players learned in serious games: In situ data collection, information trails, and quantitative analysis. In Q. Mehdi (Ed.), Proceedings of the Computer Games: AI, Animation, Mobile, Educational & Serious Games Conference, Louiseville (CGAMES 2007) (pp. 10-19). Wolverhampton, England: University of Wolverhampton.
50. Loh, C. S., & Byun, J. H. (2009). Modding Neverwinter Nights into serious game. In D. Gibson & Y. K. Baek (Eds.), Digital simulations for improving education: Learning through artificial teaching environments (pp. 408-426). Hershey, PA: IGI-Global.
51. Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics. Education and Information Technologies, 20(1), 5-19. doi: 10.1007/s10639-013-9263-y
52. Loh, C. S., & Sheng, Y. (2013). Performance metrics for serious games: Will the (real) expert please step forward? In Proceedings of the Computer Games: AI, Animation, Mobile, Educational & Serious Games Conference (CGAMES 2013) (pp. 202-206). Louiseville. IEEE. doi:10.1109/CGames.2013.6632633.
53. Loh, C. S., & Sheng, Y. (2014). Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computer in Human Behavior, 39, 322-330. doi:10.1016/j.chb.2014.07.022.
54. Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement. New York: Springer.
55. Loh, C. S., Sheng, Y., & Li, I.-H. (2015). Predicting expert-novice performance as Serious Games Analytics with objective-oriented and navigational action sequences. Computers in Human Behavior. 49:147-155. doi:10.1016/j.chb.2015.02.053.
56. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Boston, MA: Addison-Wesley.
57. Medler, B., & Magerko, B. (2011). Analytics of play: Using information visualization and gameplay practices for visualizing video game data. Parsons Journal for Information Mapping, 3(1), 1-12.
58. Michael, D., & Chen, S. (2006). Serious games: Games that educate, train, and inform. Boston: Thomson Course Technology PTR.
59. Mislevy, R. J. (1985). Estimation of latent group effects. Journal of the American Statistical Association, 80(392), 993-997. doi:10.1080/01621459.1985.10478215.
60. Moura, D., Seif El-Nasr, M., & Shaw, C. D. (2011). Visualizing and understanding players’ behavior in video games: Discovering patterns and supporting aggregation and comparison. In Proceedings of the 2011 ACM SIGGRAPH Symposium on Video Games (pp. 11-15). New York: ACM Press. doi: 10.1145/2037692.2037695.
61. Nacke, L. E., Drachen, A., & Göbel, S. (2010). Methods for evaluating gameplay experience in a serious gaming context. International Journal of Computer Science in Sport, 9(2), 1-12.
62. Niedermayer, D. (1998). An introduction to Bayesian networks and their contemporary applications. Retrieved December 12, 2014, from http://www.niedermayer.ca/papers/bayesian/bayes.html
63. O’Rourke, E., Butler, E., Liu, Y. E., Ballweber, C., & Popovic, Z. (2013). The effects of age on player behavior in educational games. In G. N. Yannakakis & E. Aarseth (Eds.), Proceedings of the 8th International Conference on the Foundations of Digital Games (FDG 2013) (pp. 158-165). Chania, Greece: Society for the Advancement of the Science of Digital Games.
64. Onisko, A., & Druzdzel, M. J. (2013). Impact of precision of Bayesian network parameters on accuracy of medical diagnostic systems. Artificial Intelligence in Medicine, 57(3), 197-206. doi:10.1016/j.artmed.2013.01.004.
65. Paulhus, D. L. (1991). Measurement and control of response biases. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitude (pp. 17-59). San Diego, CA: Academic.
66. Pieters, R., & Warlop, L. (1999). Visual attention during brand choice: The impact of time pressure and task motivation. International Journal of Research in Marketing, 16(1), 1-16. doi: 10.1016/S0167-8116(98)00022-6.
67. Pruett, C. (2010). Hot failure: Tuning gameplay with simple player metrics. Game Developer Magazine. Retrieved December 12, 2014, from http://gamasutra.com/view/feature/6155/hot_failure_tuning_gameplay_with_.php
68. Quellmalz, E., Timms, M., & Schneider, S. (2009). Assessment of student learning in science simulations and games. In processings of the Workshop on learning science: Computer games, simulations, and education. Washington, DC: National Academy of Sciences.
69. Robinson, R. W. (2007). Learning the structure of Bayesian networks. In T. D. Nielsen & F. V. Jensen (Eds.), Bayesian networks and decision graphs (2nd ed., pp. 229-264). New York: Springer.
70. Roese, N. J., & Jamieson, D. W. (1993). Twenty years of bogus pipeline research: A critical review and meta-analysis. Psychological Bulletin, 114, 363-375.
71. Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning and Assessment, 8(4), 3-41.
72. Sandford, R., & Williamson, B. (2005). Games and learning: A handbook from Futurelab. Bristol, England: Futurelab.
73. Sawyer, B., & Rejeski, D. (2002). Serious games: Improving public policy through game-based learning and simulation. Washington, DC: Woodrow Wilson International Center for Scholars. Retrieved December 12, 2014, from http://www.seriousgames.org/images/seriousarticle.pdf
74. Scarlatos, L. L., & Scarlatos, T. (2010). Visualizations for the assessment of learning in computer games. In Proceedings of the 7th International Conference & Expo on Emerging Technologies for a Smarter World, Incheon, S. Korea (CEWIT 2010). Retrieved December 12, 2014, from http://ms.cc.sunysb.edu/~lscarlatos/pubs/LLS2010_CEWIT.pdf
75. Seif El-Nasr, M., Drachen, A., & Canossa, A. (Eds.). (2013). Game analytics: Maximizing the value of player data. London: Springer.
76. Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503-524). Charlotte, NC: Information Age.
77. Shute, V. J., Masduki, I., Donmez, O., Dennen, V. P., Kim, Y. J., Jeong, A. C., et al. (2010). Modeling, assessing, and supporting key competencies within game environments. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (Part 4) (pp. 281-309). Boston: Springer. doi:10.1007/978-1-4419-5662-0_15.
78. Shute, V. J., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video games. Cambridge, England: MIT Press.
79. Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295-321). New York: Routledge.
80. Smith, S. P., & Du’Mont, S. (2009). Measuring the effect of gaming experience on virtual environment navigation tasks. In K. Kiyokawa, S. Coquillart, & R. Balakrishnan (Eds.), Proceedings of the IEEE Symposium on 3D User Interfaces (3DUI 2009) (pp. 3-10). Louisiana: IEEE. doi: 10.1109/3DUI.2009.4811198.
81. Thawonmas, R., Ho, J.Y., & Matsumoto, Y. (2003). Identification of player types in massively multiplayer online games. In Proceedings of the 34th annual conference of international simulation and gaming association, Chiba, Japan (ISAGA 2003) (pp. 893-900).
82. Thawonmas, R., & Iizuka, K. (2008). Visualization of online-game players based on their action behaviors. International Journal of Computer Games Technology, 2008, 1-9. doi:10.1155/2008/906931.
83. Torrente, J., Borro-Escribano, B., Freire, M., del Blanco, A., Marchiori, E. J., Martinez-Ortiz, I., et al. (2014). Development of game-like simulations for procedural knowledge in healthcare education. IEEE Transactions on Learning Technologies, 7(1), 69-82. doi:10.1109/TLT.2013.35.
84. Underwood, J. (2005). Novice and expert performance with a dynamic control task: Scanpaths during a computer game. In G. Underwood (Ed.), Cognitive processes in eye guidance (pp. 303-323). Oxford, England: Oxford University Press. doi:10.1093/acprof:oso/9780198566816.003.0013.
85. Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. EDUCAUSE Review, 41(2), 16-30.
86. Wallner, G. (2013). Play-graph: A methodology and visualization approach for the analysis of gameplay data. In G. N. Yannakakis & E. Aarseth (Eds.), Proceedings of the 8th International Conference on the Foundations of Digital Games (FDG 2013) (pp. 253-260). Crete, Greece: Society for the Advancement of the Science of Digital Games.
87. Wallner, G., & Kriglstein, S. (2012). A spatiotemporal visualization approach for the analysis of gameplay data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12) (pp. 1115-1124). New York: ACM Press. doi:10.1145/2207676.2208558
88. Wallner, G., & Kriglstein, S. (2013). Visualization-based analysis of gameplay data—A review of literature. Entertainment Computing, 4(3), 143-155. doi:10.1016/j.entcom.2013.02.002.
89. Wickens, C. D., Stokes, A., Barnett, B., & Hyman, F. (1993). The effects of stress on pilot judgment in a MIDIS simulator. In O. Svenson & A. J. Maule (Eds.), Time pressure and stress in human judgment and decision making (pp. 271-292). Boston: Springer. doi:10.1007/978-1-4757-6846-6_18.
90. Williams, A. M., & Ford, P. R. (2008). Expertise and expert performance in sport. International Review of Sport and Exercise Psychology, 1(1), 4-18. doi:10.1080/17509840701836867.
91. Young, M. E., Sutherland, S. C., & Cole, J. J. (2011). Individual differences in causal judgment under time pressure: Sex and prior video game experience as predictors. International Journal of Comparative Psychology, 24(1), 76-98.