Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games

Computers in Human Behavior

Volumn 39, Issue , 2014, Pages 322-330

  Loh, Christian Sebastian ^a   Sheng, Yanyan ^a  

^a SOUTHERN ILLINOIS UNIVERSITY   (United States)

Author keywords

Decision making; Expert novice performance; Human computer interactions; Metrics and methodologies; Similarity index

Indexed keywords

BEHAVIORAL RESEARCH; DECISION MAKING; HUMAN COMPUTER INTERACTION; COMPUTER AIDED INSTRUCTION;

APPROPRIATE METRICS; CORRECT SOLUTION; EXPERT-NOVICE PERFORMANCE; LEARNING ENVIRONMENTS; METRICS AND METHODOLOGIES; PERFORMANCE MEASUREMENTS; SIMILARITY INDICES; STRING SIMILARITY; MULTIPLE SOLUTIONS; NOVICE PERFORMANCE;

COMPUTER AIDED INSTRUCTION; HUMAN COMPUTER INTERACTION;

EID: 84907307998     PISSN: 07475632     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.chb.2014.07.022     Document Type: Article

References (53)

1. Alvarez, J., Djaouti, D., Rampnoux, O., & Alvarez, V. (2011). Serious games market: Some key figures (from 1950's to 2000's). < http://serious.gameclassification.com/files/articles/sgc-report-03-11.pdf > Retrieved 22.10.11.
2. F. Baudoin, J.E. Sawyer, and C. Avellone Neverwinter Nights 2 2009 Atari Los Angeles, CA
3. F. Bellotti, B. Kapralos, K. Lee, and P. Moreno-Ger User assessment in serious games and technology-enhanced learning Advances in Human-Computer Interaction 2013 2013 2 10.1155/2013/120791
4. H. Ben Zur, and S.J. Breznitz The effect of time pressure on risky choice behavior Acta Psychologica 47 2 1981 89 104 10.1016/0001-6918(81)90001-9
5. W.R. Boot, A.F. Kramer, D.J. Simons, M. Fabiani, and G. Gratton The effects of video game playing on attention, memory, and executive control Acta Psychologica 129 3 2008 387 398 10.1016/j.actpsy.2008.09.005
6. A. Canossa, M. Seif El-Nasr, and A. Drachen Benefits of game analytics: Stakeholders, contexts and domains M. Seif El-Nasr, A. Drachen, A. Canossa, Game analytics: Maximizing the value of player data 2013 Springer-Verlag London, UK 41 52 10.1007/978-1-4471-4769-5
7. A. De Groot Thought and choice in chess 2nd ed. 1978 University Press Cambridge, UK
8. A. Drachen, M. Seif El-Nasr, and A. Canossa Game analytics - The basics M. Seif El-Nasr, A. Drachen, A. Canossa, Game analytics: Maximizing the value of player data 2013 Springer-Verlag London, UK 13 40 10.1007/978-1-4471-4769-5-2
9. S.E. Dreyfus The five-stage model of adult skill acquisition Bulletin of Science, Technology and Society 24 3 2004 177 181 10.1177/0270467604264992
10. Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition (p. 18). Berkeley, CA. < http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc. pdf&AD=ADA084551 >.
11. K.A. Ericsson, and N. Charness Expert performance: Its structure and acquisition American Psychologist 49 8 1994 725 747
12. K.A. Ericsson, N. Charness, P.J. Feltovich, and R.R. Hoffman The Cambridge handbook of expertise and expert performance. (p. 901) 2006 Cambridge University Press New York, NY
13. K.A. Ericsson, M.J. Prietula, and E.T. Cokely The making of an expert Harvard Business Review 85 7-8 2007 114 121 < http://www.ncbi.nlm.nih.gov/ pubmed/17642130q >
14. I. Gauthier, P. Williams, M.J. Tarr, and J.W. Tanaka Training "Greeble" experts: A framework for studying expert object recognition processes Vision Research 38 1998 2401 2428 10.1016/S0042-6989(97)00442-2
15. D. Ifenthaler, D. Eseryel, and X. Ge Assessment in Game-based Learning: Foundations, Innovations, and Perspectives. (p. 465) 2012 Springer New York, NY
16. P. Jaccard The distribution of the flora in the Alpine Zone New Phytologist 11 2 1912 37 50 10.1111/j.1469-8137.1912.tb05611.x
17. J. Kirkley, S. Kirkley, and J. Heneghan Building bridges between serious game design and instructional design B.E. Shelton, D.A. Wiley, Educational design & use of computer simulation games 2007 Sense Publishers Rotterdam, The Netherlands 59 81
18. J. Kjeldskov, and M.B. Skov Studying usability in sitro: Simulating real world phenomena in controlled environments International Journal of Human-Computer Interaction 22 1 & 2 2007 7 36 10.1080/10447310709336953
19. Kozlov, S., & Reinhold, N. (2007). To play or not to play: Can companies learn to be n00bs, LFG, and lvl-up? In Proceedings of IR 8.0 - Let's Play!, 8th Association of Internet Researchers Conference. Vancouver, Canada.
20. O.E. Krigolson, L.J. Pierce, C.B. Holroyd, and J.W. Tanaka Learning to become an expert: Reinforcement learning and the acquisition of perceptual expertise Journal of Cognitive Neuroscience 21 9 2009 1834 1841 10.1162/jocn.2009.21128
21. P.A. Lachenbruch, and M.R. Mickey Estimation of error rates in discriminant analysis Technometrics 10 1 1968 1 11 10.1080/00401706.1968. 10490530
22. C.S. Loh Designing online games assessment as "information trails V. Sugumaran, Intelligent information technologies: Concepts, methodologies, tools and applications 2007 Idea Group Inc. Hershey, PA 553 574
23. Loh, C. S. (2011). Using in situ data collection to improve the impact and return of investment of game-based learning. In Proceedings of the 61st international council for educational media and the XIII International Symposium on Computers in Education (ICEM&SIIE'2011) joint conference (pp. 801-811). Aveiro, Portugal.
24. C.S. Loh Improving the impact and return of investment of game-based learning International Journal of Virtual and Personal Learning Environments 4 1 2012 1 15 10.4018/jvple.2013010101
25. C.S. Loh Information trails: In-process assessment of game-based learning D. Ifenthaler, D. Eseryel, X. Ge, Assessment in game-based learning: Foundations, innovations, and perspectives 2012 Springer New York, NY 123 144 10.1007/978-1-4614-3546-4
26. C.S. Loh, A. Anantachai, J.H. Byun, and J. Lenox Assessing what players learned in serious games: In situ data collection, Information Trails, and quantitative analysis Q. Mehdi, Proceedings of the computer games: AI, animation, mobile, educational & serious games conference (CGAMES 2007) 2007 University of Wolverhampton Wolverhampton, UK
27. Loh, C. S., & Sheng, Y. (2013). Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics. Education and Information Technologies, (Online First). http://dx.doi.org/10.1007/s10639-013- 9263-y.
28. C.S. Loh, Y. Sheng, and D. Ifenthaler Serious Games analytics: Methodologies for performance measurement, assessment, and improvement with Serious Games 2015 Springer New York, NY Manuscript in preparation
29. B. Medler Player dossiers: Analyzing gameplay data as a reward Game Studies 11 1 2011 < http://gamestudies.org/1101/articles/medler >
30. Michael, D., & Chen, S. (2005). Proof of Learning: Assessment in Serious Games. Gamasutra. < http://www.gamasutra.com/view/feature/2433/proof- of-learning-assessment-in-.php > Retrieved 22.11.12.
31. D. Michael, and S. Chen Serious games: Games that educate, train, and inform 2006 Thomson Course Technology PTR Boston, MA
32. Monge, A. E., & Elkan, C. P. (1997). An efficient domain-independent algorithm for detecting approximately duplicate database records. In Proceedings of the SIGMOD 1997 Workshop on Research Issues on Data Mining and Knowledge Discovery (pp. 23-29). Tuscon, AZ.
33. D. Moura, M. Seif El-Nasr, and C.D. Shaw Visualizing and understanding players' behavior in video games: Discovering patterns and supporting aggregation and comparison Proceedings of the 2011 ACM SIGGRAPH symposium on video games 2011 ACM Press New York, NY 11 15 10.1145/2037692.2037695
34. Nisen, M. (2013, May 6). Moneyball at work: They've discovered what really makes a great employee. Business Insider. New York, NY. < http://www.businessinsider.com/big-data-in-the-workplace-2013-5 >.
35. R. Pieters, and L. Warlop Visual attention during brand choice: The impact of time pressure and task motivation International Journal of Research in Marketing 16 1 1999 1 16 10.1016/S0167-8116(98)00022-6
36. R. Rao From unstructured data to actionable intelligence IT Professional 5 6 2003 29 35 10.1109/MITP.2003.1254966
37. C. Romero, S. Ventura, M. Pechenizkiy, and R.S.J.d. Baker The handbook of educational data mining 1st ed. 2010 CRC Press Boca Raton, FL 535
38. B.H. Rosenberg, D. Landsittel, and T.D. Averch Can video games be used to predict or improve laparoscopic skills? Journal of Endourology 19 3 2005 372 376 10.1089/end.2005.19.372
39. H. Sabri, B. Cowan, B. Kapralos, M. Porte, D. Backstein, and A. Dubrowskie Serious games for knee replacement surgery procedure education and training Procedia - Social and Behavioral Sciences 2 2 2010 3483 3488 10.1016/j.sbspro.2010.03.539
40. J. Sauro, and E. Kindlund A method to standardize usability metrics into a single score Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '05 (p. 401) 2005 ACM Press New York, NY 10.1145/1054972.1055028
41. R.A. Schmidt, and T. Lee Motor control and learning: A behavioral emphasis 5th ed. 2011 Human Kinetics Champaign, IL
42. V.J. Shute, I. Masduki, O. Donmez, V.P. Dennen, Y.-J. Kim, and A.C. Jeong et al. Modeling, assessing, and supporting key competencies within game environments D. Ifenthaler, P. Pirnay-Dummer, N.M. Seel, Computer-based diagnostics and systematic analysis of knowledge (Part 4) 2010 Springer, US Boston, MA 281 309
43. Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S. B., Ferguson, R., et al. (2011). Open learning analytics: An integrated & modularized platform. < http://solaresearch.org/OpenLearningAnalytics.pdf >.
44. J.W. Tanaka, T. Curran, and D.L. Sheinberg The training and transfer of real-world perceptual expertise Psychological Science 16 2 2005 145 151 10.1111/j.0956-7976.2005.00795.x
45. R. Thawonmas, and K. Iizuka Visualization of online-game players based on their action behaviors International Journal of Computer Games Technology 2008 2008 1 9 10.1155/2008/906931
46. A. Tilinger, and C. Sik-Lanyi Issues of hand preference in computer presented information and virtual realities G. Ghinea, S.Y. Chen, Digital multimedia perception and design 2006 IGI Global Hershey, PA 224 243 10.4018/978-1-59140-860-4
47. J. Underwood Novice and expert performance with a dynamic control task: Scanpaths during a computer game G. Underwood, Cognitive processes in eye guidance 2005 Oxford University Press Oxford 303 323 10.1093/acprof:oso/ 9780198566816.003. 0013
48. R. Van Eck Digital game-based learning: It's not just the digital natives who are restless... EDUCAUSE Review 41 2 2006 16 30
49. Wallner, G. (2013). Play-Graph: A methodology and visualization approach for the analysis of gameplay data. In G. N. Yannakakis & E. Aarseth (Eds.), 8th International Conference on the Foundations of Digital Games (FDG 2013) (pp. 253-260). Crete, Greece: Society for the Advancement of the Science of Digital Games. < http://www.fdg2013.org/program/papers/paper33-wallner.pdf >.
50. G. Wallner, and S. Kriglstein A spatiotemporal visualization approach for the analysis of gameplay data SIGCHI conference on human factors in computing systems (CHI '12) 2012 ACM Press New York, NY 1115 1124 10.1145/2207676.2208558
51. C.D. Wickens, A. Stokes, B. Barnett, and F. Hyman The effects of stress on pilot judgment in a MIDIS Simulator O. Svenson, A.J. Maule, Time pressure and stress in human judgment and decision making 1993 Springer, US Boston, MA 271 292 10.1007/978-1-4757-6846-6-18
52. Winkler, W. E. (1999). The state of record linkage and current research problems (p. 15). Washington, DC. http://www.census.gov/srd/papers/pdf/rr99- <04.pdf >.
53. M.E. Young, S.C. Sutherland, and J.J. Cole Individual differences in causal judgment under time pressure: Sex and prior video game experience as predictors International Journal of Comparative Psychology 24 1 2011 76 98 < http://www.comparativepsychology.org/ijcp-2011-1/5.Young-etal-PDF.pdf >