Predicting expert-novice performance as serious games analytics with objective-oriented and navigational action sequences

Computers in Human Behavior

Volumn 49, Issue , 2015, Pages 147-155

  Loh, Christian Sebastian ^a   Sheng, Yanyan ^a   Li, I Hung ^a  

^a SOUTHERN ILLINOIS UNIVERSITY   (United States)

Author keywords

Action sequences; Discriminant analysis; Expert novice performance; Partial least squares; Serious games analytics; Similarity measures

Indexed keywords

DISCRIMINANT ANALYSIS; LEAST SQUARES APPROXIMATIONS;

ACTION SEQUENCES; NOVICE PERFORMANCE; PARTIAL LEAST SQUARE (PLS); SERIOUS GAMES; SIMILARITY MEASURE;

CODES (SYMBOLS);

EID: 84924856407     PISSN: 07475632     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.chb.2015.02.053     Document Type: Article

References (72)

1. Adar, E., Teevan, J., Dumais, S. T. (2008). Large scale analysis of web revisitation patterns. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1197-1206). New York, NY: ACM. http://dx.doi.org/10.1145/1357054.1357241.
2. M.A. Awad, and I. Khalil Prediction of user's web-browsing behavior: Application of markov model IEEE Transactions on Systems, Man, Cybernetics, Part B: Cybernetics 42 4 2012 1131 1142 10.1109/TSMCB.2012.2187441
3. D. Ballabio, and V. Consonni Classification tools in chemistry. Part 1: Linear models. PLS-DA Analytical Methods 5 16 2013 3790 3798
4. S.L. Beilock, and T.H. Carr From novice to expert performance: Memory, attention and the control of complex sensori-motor skills A.M. Williams, N.J. Hodges, Skill acquisition in sport: Research, theory and practice 1st ed. 2004 Routledge New York, NY 309 327
5. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: An overview. Advances in Human-Computer Interaction, 11. < http://www.hindawi.com/journals/ahci/2013/136864/ > (Retrieved 12.12.14).
6. H. Ben Zur, and S.J. Breznitz The effect of time pressure on risky choice behavior Acta Psychologica 47 2 1981 89 104 10.1016/0001-6918(81)90001-9
7. W.R. Boot, D.P. Blakely, and D.J. Simons Do action video games improve perception and cognition? Frontiers in Psychology 2 2011 226
8. W.R. Boot, A.F. Kramer, D.J. Simons, M. Fabiani, and G. Gratton The effects of video game playing on attention, memory, and executive control Acta Psychologica 129 3 2008 387 398 10.1016/j.actpsy.2008.09.005
9. G.E.P. Box, and N.R. Draper Empirical model building and response surfaces 1987 John Wiley & Sons New York, NY
10. W.L. Bryan, and N. Harter Studies on the telegraphic language: The acquisition of a hierarchy of habits Psychological Review 6 1899 345 375
11. C. Cappiello, M. Matera, M. Picozzi, G. Sprega, D. Barbagallo, and C. Francalanci DashMash: A mashup environment for end user development S. Auer, O. Díaz, G.A. Papadopoulos, Web Engineering Vol. 6757 2011 Springer Berlin, Heidelberg 152 166 10.1007/978-3-642-22233-7
12. Chi, E. H., Pirolli, P., Chen, K., Pitkow, J. (2001). Using information scent to model user information needs and actions and the Web. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI '01) (pp. 490-497). New York, NY: ACM Press. http://dx.doi.org/10.1145/365024.365325.
13. R.E. Clark Confounding in educational computing research Journal of Educational Computing Research 1 2 1985 137 148
14. R.E. Clark Dangers in the evaluation of instructional media Academic Medicine 67 12 1992 819 820 < http://journals.lww.com/academicmedicine/Fulltext/1992/12000/Dangers-in-the-evaluation-of-instructional-media-.4.aspx > (Retrieved 12.12.14).
15. R.E. Clark Media will never influence learning Educational Technology Research and Development 42 2 1994 21 29 10.1007/BF02299088
16. D. Crookall Serious games, debriefing, and simulation/gaming as a discipline Simulation and Gaming 41 6 2010 898 920 10.1177/1046878110390784
17. A. de Groot Thought and Choice in Chess 2nd ed. 1978 University Press Cambridge, UK
18. A. Drachen, and A. Canossa Evaluating motion: Spatial user behaviour in virtual environment International Journal of Arts and Technology 4 3 2011 294 314 10.1504/IJART.2011.041483
19. Drachen, A., Thurau, C., Togelius, J., Yannakakis, G. N., & Bauckhage, C. (2013). Game data mining. In: M. Seif El-Nasr, A. Drachen, A. Canossa (Eds.), Game analytics: Maximizing the value of player data. London, UK: Springer London
20. S.E. Dreyfus The five-stage model of adult skill acquisition Bulletin of Science, Technology and Society 24 3 2004 177 181 10.1177/0270467604264992
21. Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition (p. 18). Berkeley, CA. < http://www.dtic.mil/get-tr-doc/pdf?AD=ADA084551 > (Retrieved 01.12.14).
22. H.L. Dreyfus, and S.E. Dreyfus Peripheral vision: Expertise in real world contexts Organization Studies 26 5 2005 779 792 10.1177/0170840605053102
23. B. Ellis Real-time analytics: Techniques to analyze and visualize streaming data 2014 John Wiley & Sons Indianapolis, IN
24. K.A. Ericsson, N. Charness, P.J. Feltovich, and R.R. Hoffman The Cambridge handbook of expertise and expert performance: Cambridge Handbooks in Psychology 2006 Cambridge University Press New York, NY
25. K.A. Ericsson, M.J. Prietula, and E.T. Cokely The making of an expert Harvard Business Review 85 7-8 2007 114 121 193
26. X. Fan, B. Miller, K.-E. Park, B.W. Winward, M. Christensen, and H.D. Grotevant An exploratory study about inaccuracy and invalidity in adolescent self-report surveys Field Methods 18 3 2006 223 244 10.1177/152822X06289161
27. P.M. Fitts, and M.I. Posner Human performance 1967 Brooks/Cole Belmont, CA
28. M. Frese, and D. Zapf Actions as the core of work psychology: A German approach 2nd ed. H.C. Triandis, M.D. Dunnette, L.M. Hough, Handbook of industrial and organizational psychology Vol. 4. 1994 Consulting Psychologists Press Paolo Alto, CA 271 340
29. I. Gauthier, P. Williams, M.J. Tarr, and J.W. Tanaka Training "Greeble" experts: A framework for studying expert object recognition processes Vision Research 38 1998 2401 2428 10.1016/S0042-6989(97)00442-2
30. C. Gonzalez Decision support for real-time, dynamic decision-making tasks Organizational Behavior and Human Decision Processes 96 2 2005 142 154 10.1016/j.obhdp.2004.11.002
31. Gonzalez, C., & Golenbock, J. (2003). Impact of numerical and graphical formats on dynamic decision making performance: An eye-tracking study. In Proceedings of the Latin American conference on human-computer interaction (CLIHC '03) (pp. 231-234). ACM. < http://dl.acm.org/citation.cfm?id=944519.944546 > (Retrieved 20.01.15).
32. N.B. Hastings, and M.W. Tracey Does media affect learning: Where are we now? TechTrends 49 2 2004 28 30 10.1007/BF02773968
33. Herder, E. (2003). Revisitation patterns and disorientation. German workshop on adaptivity and user modeling in interactive systems (ABIS '03) (pp. 291-294). Karlsruhe, Germany. < http://km.aifb.uni-karlsruhe.de/ws/LLWA/abis/herder.pdf > (Retrieved 12.12.14).
34. C. Hölscher, and G. Strube Web Search behavior of Internet experts and newbies Computer Networks 33 1 2000 337 346 10.1016/S1389-1286(00)00031-1
35. J.C. Hong, and M.C. Liu A study on thinking strategy between experts and novices of computer games Computers in Human Behavior 19 2 2003 245 258 10.1016/S0747-5632(02)00013-4
36. K. Hornbæk Current practice in measuring usability: Challenges to usability studies and research International Journal of Human-Computer Studies 64 2 2006 79 102 10.1016/j.ijhcs.2005.06.002
37. Hoskin, R. (2012). The dangers of self-report. < http://www.sciencebrainwaves.com/uncategorized/the-dangers-of-self-report/ > (Retrieved 12.12.14).
38. D. Ifenthaler Bridging the gap between expert-novice differences Journal of Research on Technology in Education 43 2 2010 103 117 10.1080/15391523.2010.10782564
39. Kozlov, S., & Reinhold, N. (2007). To play or not to play: Can companies learn to be n00bs, LFG, and lvl-up? In Proceedings of IR 8.0 - Let's Play! 8th association of internet researchers conference. Vancouver, Canada.
40. R.B. Kozma, and J. Russell Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena Journal of Research in Science Teaching 34 9 1997 949 968
41. O.E. Krigolson, L.J. Pierce, C.B. Holroyd, and J.W. Tanaka Learning to become an expert: Reinforcement learning and the acquisition of perceptual expertise Journal of Cognitive Neuroscience 21 9 2009 1834 1841 10.1162/jocn.2009.21128
42. Krulak, C. (1997). Military thinking and decision making exercises (No. 1500.55). Washington, DC. < http://www.marines.mil/Portals/59/Publications/MCO1500.55.pdf > (Retrieved 01.12.14).
43. Law, B., Atkins, M. S., Kirkpatrick, A. E., & Lomax, A. J. (2004). Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. In Proceedings of the 2004 symposium on eye tracking research and applications (ETRA '04) (pp. 41-48). New York, NY: ACM Press. http://dx.doi.org/10.1145/968363.968370.
44. C.S. Loh Improving the impact and return of investment of game-based learning International Journal of Virtual and Personal Learning Environments 4 1 2012 1 15 10.4018/jvple.2013010101
45. Loh, C. S., & Sheng, Y. (in press). Assessing expert performance as serious games analytics: From data to insights. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement. New York, NY: Springer.
46. C.S. Loh Information trails: In-process assessment of game-based learning D. Eseryel, D. Ifenthaler, X. Ge, Assessment in game-based learning: Foundations, innovations, and perspectives 2012 Springer New York, NY 123 144 10.1007/978-1-4614-3546-4
47. Loh, C. S., Anantachai, A., Byun, J. H., & Lenox, J. (2007). Assessing what players learned in serious games: In situ data collection, information trails, and quantitative analysis. In Q. Mehdi (Ed.), Proceedings of the computer games: AI, animation, mobile, educational and serious games conference (CGAMES 2007) (pp. 10-19). Wolverhampton, UK: University of Wolverhampton.
48. Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.) (in press). Serious games analytics: Methodologies for human performance measurement, assessment, and improvement. New York, NY: Springer.
49. C.S. Loh Designing online games assessment as "Information Trails" D. Gibson, C. Aldrich, M. Prensky, Games and simulation in online learning: Research and development frameworks 2006 Idea Group Inc. Hershey, PA 323 348
50. C.S. Loh, and Y. Sheng Performance metrics for serious games: Will the (real) expert please step forward? Proceedings of CGAMES'2013 USA 2013 IEEE 202 206 10.1109/CGames.2013.6632633
51. C.S. Loh, and Y. Sheng Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games Computer in Human Behavior 39 2014 322 330 10.1016/j.chb.2014.07.022
52. C.S. Loh, and Y. Sheng Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics Education and Information Technologies 20 1 2015 5 19 10.1007/s10639-013-9263-y forthcoming
53. Michael, D., & Chen, S. (2005). Proof of learning: Assessment in serious games. < http://www.gamasutra.com/view/feature/2433/proof-of-learning-assessment-in-.php > (Retrieved 22.09.12).
54. J. Mishra, M. Zinni, D. Bavelier, and S.A. Hillyard Neural basis of superior performance of action videogame players in an attention-demanding task The Journal of Neuroscience 31 3 2011 992 998 10.1523/jneurosci.4834-10.2011
55. R. Pieters, and L. Warlop Visual attention during brand choice: The impact of time pressure and task motivation International Journal of Research in Marketing 16 1 1999 1 16 10.1016/S0167-8116(98)00022-6
56. M. Rauterberg From novice to expert decision behaviour: A qualitative modelling approach with Petri Nets Advances in Human Factors/Ergonomics 20 1995 449 454 10.1016/S0921-2647(06)80257-1
57. B.H. Rosenberg, D. Landsittel, and T.D. Averch Can video games be used to predict or improve laparoscopic skills? Journal of Endourology 19 3 2005 372 376 10.1089/end.19.372
58. H. Sabri, B. Cowan, B. Kapralos, M. Porte, D. Backstein, and A. Dubrowskie Serious games for knee replacement surgery procedure education and training Procedia - Social and Behavioral Sciences 2 2 2010 3483 3488 10.1016/j.sbspro.2010.03.539
59. M. Seif El-Nasr, A. Drachen, and A. Canossa Game analytics: Maximizing the value of player data 2013 Springer London, UK
60. Smith, S. P., Blackmore, K., & Nesbitt, K. (in press). A meta-analysis of data collection in serious games research. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for human performance measurement, assessment, and improvement. New York, NY: Springer.
61. J.W. Tanaka, T. Curran, and D.L. Sheinberg The training and transfer of real-world perceptual expertise Psychological Science 16 2 2005 145 151 10.1111/j.0956-7976.2005.00795.x
62. Thawonmas, R., Yoshida, K., Lou, J. -K., & Chen, K. -T. (2009). Analysis of area revisitation patterns in World of Warcarft. In S. Natkin & J. Dupire (Eds.), In Proceedings of the 8th international conference of entertainment computing (ICEC 2009) (Vol. 5709, pp. 13-23). Springer Berlin Heidelberg. http://dx.doi.org/10.1007/978-3-642-04052-8.
63. R. Thawonmas, K. Yoshida, J.-K. Lou, and K.-T. Chen Analysis of revisitations in online games Entertainment Computing 2 4 2011 215 221 10.1016/j.entcom.2011.01.003
64. J. Underwood Novice and expert performance with a dynamic control task: Scanpaths during a computer game G. Underwood, Cognitive processes in eye guidance 2005 Oxford University Press Oxford 303 323 10.1093/acprof:oso/9780198566816.003.0013
65. Weber, B. G., & Mateas, M. (2009). A data mining approach to strategy prediction. In IEEE symposium on computational intelligence and games (CIG 2009) (pp. 140-147). Milano: IEEE. http://dx.doi.org/10.1109/CIG.2009.5286483.
66. C.D. Wickens, A. Stokes, B. Barnett, and F. Hyman The effects of stress on pilot judgment in a MIDIS simulator O. Svenson, A.J. Maule, Time pressure and stress in human judgment and decision making 1993 Springer Boston, MA, US 271 292 10.1007/978-1-4757-6846-6-18
67. S. Wiedenbeck, V. Fix, and J. Scholtz Characteristics of the mental representations of novice and expert programmers: An empirical study International Journal of Man-Machine Studies 39 5 1993 793 812 10.1006/imms.1993.1084
68. A.M. Williams Perceptual skill in soccer: Implications for talent identification and development Journal of Sports Sciences 18 9 2000 737 750 10.1080/02640410050120113
69. A.M. Williams, and K.A. Ericsson Perceptual-cognitive expertise in sport: Some considerations when applying the expert performance approach Human Movement Science 24 3 2005 283 307 10.1016/j.humov.2005.06.002
70. Winkler, W. E. (2006). Overview of Record Linkage and current research directions (pp. 1-44). Washington, DC. < http://www.census.gov/srd/papers/pdf/rrs2006-02.pdf > (Retrieved 01.12.14).
71. M.E. Young, S.C. Sutherland, and J.J. Cole Individual differences in causal judgment under time pressure: Sex and prior video game experience as predictors International Journal of Comparative Psychology 24 1 2011 76 98 < http://www.comparativepsychology.org/ijcp-2011-1/5.Young-etal-PDF.pdf > (Retrieved 12.12.14)
72. G. Zoeller Game development telemetry in production M. Seif El-Nasr, A. Drachen, A. Canossa, Game analytics: Maximizing the value of player data 2013 Springer London, UK 111 136