Continuous recognition of player's affective body expression as dynamic quality of aesthetic experience

IEEE Transactions on Computational Intelligence and AI in Games

Volumn 4, Issue 3, 2012, Pages 199-212

  Savva, Nikolaos ^a,b   Scarinzi, Alfonsina ^c,d   Bianchi Berthouze, Nadia ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b Hydro Comp Enterprises ^*  (Cyprus)

^c UNIVERSITY OF GÖTTINGEN   (Germany)

^d UNIVERSITY OF GENOA   (Italy)

Author keywords

Aesthetic experience; automatic recognition of emotion; embodied interaction; engagement; full body computer game

Indexed keywords

AESTHETIC EXPERIENCE; AUTOMATIC RECOGNITION; COMPUTER GAME; EMBODIED INTERACTION; ENGAGEMENT;

HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER GRAPHICS;

COMPUTER SOFTWARE;

EID: 84866512354     PISSN: 1943068X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCIAIG.2012.2202663     Document Type: Article

References (75)

1. M. G. Petersen, O. S. Iversen, and P. G. Krogh, "Aesthetic interaction- A pragmatist's aesthetics of interactive systems," in Proc. Designing Interactive Syst., 2004, pp. 269-276.
2. J. H. Kim, D. V. Gunn, E. Schuh, B. Phillips, R. J. Pagulayan, and D. Wixon, "Tracking real-time user experience (TRUE): A comprehensive instrumentation solution for complex systems," in Proc. 26th Annu. SIGCHI Conf. Human Factors Comput. Syst., 2008, pp. 443-452.
3. J. L. Elman, "Finding structure in time," Cogn. Sci., vol. 14, pp. 179-211, 1990.
4. J. H. Kim, D. V. Gunn, E. Schuh, B. Phillips, R. J. Pagulayan, and D. Wixon, "Tracking real-time user experience (TRUE): A comprehensive instrumentation solution for complex systems," in Proc. 26th Annu. SIGCHI Conf. Human Factors Comput. Syst., 2008, pp. 443-452.
5. F. Muller and N. Bianchi-Berthouze, Evaluating Exertion Games Experiences from Investigating Movement Based, ser. Human-Computer Interaction. New York: Springer-Verlag, 2010, pt. 4, pp. 187-207.
6. N. Bianchi-Berthouze, "Understanding the role of body movement in player engagement," Human-Computer Interaction [Online]. Available: http://web4.cs.ucl.ac.uk/uclic/people/n.berthouze/BerthouzeHCI12.pdf, to be published
7. K. Isbister, U. Schwekendiek, and J. Frye, "Wriggle: An exploration of emotional and social effects of movement," in Proc. 26th Annu. SIGCHI Conf. Human Factors Comput. Syst., 2011, pp. 1885-1890.
8. A. Melzer, I. Derks, J. Heydekorn, and G. Steffgen, "Click or strike: Realistic versus standard game controls in violent video games and their effects on aggression," in Entertainment Computing, ser. Lecture Notes in Computer Science,H. S. Yang, R.Malaka, J. Hoshino, and J. H. Han, Eds. Berlin, Germany: Springer-Verlag, 2010, vol. 6243, pp. 171-182.
9. J. H. Brockmyer, C. M. Fox, K. A. Curtiss, E. McBroom, K. M. Burkhart, and J. N. Pidruzny, "The development of the game engagement questionnaire:Ameasure of engagement in video game-playing," J. Exp. Social Psychol., vol. 45, no. 4, pp. 624-634, 2009.
10. J. Hoysniemi, "International survey on the dance revolution game," Comput. Entertain. Sec. Games, User Interface and Performing Arts, vol. 4, no. 2,8, 2006.
11. M. Slater, A. Steed, J. McCarthy, and F. Maringelli, "The influence of body movement on subjective presence in virtual environments," Human Factors, vol. 40, pp. 469-477, 1998. (Pubitemid 28551225)
12. M. Slater, M. Usoh, and A. Steed, "Taking steps: The influence of a walking metaphor on presence in virtual reality," ACM Trans. Computer-Human Interaction, vol. 2, pp. 201-219, 1995.
13. N. Lazzaro, "Why we play games: Four keys to more emotion without story," XEO Design Inc., Tech. Rep., 2004.
14. M. G. Petersen, "Aesthetics of interaction-A pragmatist perspective," in Proc. Workshop Aesthetic Approaches Human-Computer Interaction, 2004, pp. 31-32.
15. B. Schuller, M. F. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic, "AVEC 2011-The first international audio/visual emotion challenge," in International Conference on Affective Computing and Intelligent Interaction, Part 2, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2011, vol. 6975, pp. 415-424.
16. T. Alexander, "John Dewey's theory of art, experience and nature," in The Horizons of Feeling. New York: SUNY, 1987.
17. J. Dewey, Art as Experience. New York: Perigee Books, 2005.
18. J. Chandler and N. Schwarz, "How extending your middle finger affects your perception of others: Learned movements influence concept accessibility," J. Exp. Social Psychol., vol. 45, no. 1, pp. 123-128, 2009.
19. J. D. Laird, "Self-attribution of emotion: The effects of expressive behavior on the quality of emotional experience," J. Personality Social Psychol., vol. 29, no. 4, pp. 475-486, 1974.
20. J. H. Riskind and C. C. Gotay, "Physical posture: Could it have regulatory or feedback effects on motivation and emotion?," Motivation Emotion, vol. 6, no. 3, pp. 273-298, 1982.
21. S. Stepper and F. Strack, "Proprioceptive determinants of emotional and nonemotional feelings," J. Personality Social Psychol., vol. 64, pp. 211-220, 1993.
22. G. L. Wells and R. E. Petty, "The effects of overt head movements on persuasion: Compatibility and incompatibility of responses," Basic Appl. Social Psychol., vol. 1, pp. 219-230, 1980.
23. J. T. Cacioppo, J. R. Priester, and G. G. Berntson, "Rudimentary determination of attitudes: II. Arm flexion and extension have differential effects on attitudes," J. Personality Social Psychol., vol. 65, pp. 5-17, 1993.
24. R. Neumann and F. Strack, "Approach and avoidance : The influence of proprioceptive and exteroceptive cues on encoding of affective information," J. Personality Social Psychol., vol. 79, no. 1, pp. 39-48, 2000.
25. P. M. Niedenthal, L. W. Barsalou, P. Winkielman, S. Krauth-Gruber, and F. Ric, "Embodiment in attitudes, social perception, and emotion," Personality Social Psychol. Rev., vol. 9, no. 3, pp. 184-211, 2005. (Pubitemid 41356682)
26. D. R. Carney, A. J. C. Cuddy, and A. J. Yap, "Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance," Psychol. Sci., vol. 21, no. 10, pp. 1363-1368, 2010.
27. J. Cole and B. Montero, "Affective proprioception," Janus Head, vol. 9, no. 2, pp. 299-317, 2007.
28. J.Y. Park, Y. C. Lee, B. S. Jo, and C.W. Lee, "Virtual playing ground interface using upper-body gesture recognition," in Proc. Workshop Frontiers Comput. Vis., 2011, DOI: 10.1109/FCV.2011.5739762.
29. J. Y. Park and J. H. Yii, "Gesture recognition based interactive boxing game," in Proc. Int. Conf. Intell. Comput., 2005.
30. A. Bleiweiss, D. Eshar, G. Kutliroff, A. Lerner, Y. Oshrat, and Y. Yanai, "Enhanced interactive gaming by blending full-body tracking and gesture animation," in Proc. ACM SIGGRAPH ASIA, 2010, DOI: 10.1145/1899950.1899984.
31. M. Roccetti, G. Marfia, and A. Semeraro, "A fast and robust gesture recognition system for exhibit gaming scenarios," in Proc. Int. ICST Conf. Simul. Tools Tech., 2011, pp. 343-350.
32. M. Roccetti, G. Marfia, and A. Semeraro, "Playing into the wild: A gesture-based interface for gaming in public spaces," J. Vis. Commun. Image Represent., vol. 23, no. 3, pp. 426-440, 2012.
33. K. Isbister, "Enabling social play: A framework for design and evaluation," in Evaluating User Experiences in Games: Concepts and Methods, R. Bernhaupt, Ed. London, U.K.: Springer-Verlag, 2010.
34. E. Brown and P. Cairns, "A grounded investigation of game immersion," in Proc. ACM Conf. Human Factors Comput., 2004, pp. 1297-1300.
35. L. Ermi and F. M̈ayr̈a, "Fundamental components of the gameplay experience: Analysing immersion," in Proc. DiGRA Conf. Changing Views: Worlds in Play, 2005, pp. 15-27.
36. G. N. Yannakakis and J. Hallam, "Entertainment modeling through physiology in physical play," Int. J. Human-Comput. Studies, vol. 66, no. 10, pp. 741-755, 2008.
37. M. Pasch, N. Bianchi-Berthouze, B. van Dijk, and A. Nijholt, "Movement-based sports video games: Investigating motivation and gaming experience," Entertain. Comput., vol. 9, no. 2, pp. 169-180, 2009.
38. J. Nijhar, N. Bianchi-Berthouze, and G. Boguslawski, "Does movement recognition precision affect the player experience in exertion games?," in Proc. Int. Conf. Intell. Technol. Interactive Entertain., 2012, vol. 78, pp. 73-82.
39. D. Kirsh and P. Maglio, "On distinguish epistemic from pragmatic action," Cogn. Sci., vol. 18, pp. 513-549, 1984.
40. P. P. Maglio,M. J.Wenger, and A.M. Copeland, "Evidence for the role of self-priming in epistemic action: Expertise and the effective use of memory," Acta Psychologica, vol. 127, no. 1, pp. 72-88, 2008.
41. N. Shaker, S. Asteriadis, G. N. Yannakakis, and K. Karpouzis, "A game-based corpus for analysing the interplay between game context and player experience," in Proc. Int. Conf. Affective Comput. Intell. Interaction , 2011, pp. 547-556.
42. M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience. New York: Harper Perennial, 1991.
43. J. MacGonigal, "This is not a game: Immersive aesthetics and collective play," MelbourneDAC, 2003.
44. S. Niedenthal, "What we talk about when we talk about game aesthetics," in Proc. Int. Conf. Digital Game Res. Assoc., 2009.
45. H. Sommerseth, "Exploring game aesthetics," in Proc. Int. Conf. Digital Game Res. Assoc., 2009.
46. H. G. Wallbott, "Bodily expression of emotion," Eur. J. Social Psychol., vol. 28, pp. 879-896, 1998.
47. M. Coulson, "Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence," J. Nonverbal Behav., vol. 28, pp. 117-139, 2004. (Pubitemid 39015676)
48. A. Kleinsmith, R. De Silva, and N. Bianchi-Berthouze, "Cross-cultural differences in recognizing affect from body posture," Interacting With Computers, vol. 18, no. 6, pp. 1371-1389, 2006. (Pubitemid 44709886)
49. R. De Silva and N. Bianchi-Berthouze, "Modeling human affective postures: An information theoretic characterization of posture features," J. Comput. Animat. Virtual Worlds, vol. 15, no. 3-4, pp. 269-276, 2004. (Pubitemid 41108005)
50. B. de Gelder, "Why bodies? Twelve reasons for including bodily expressions in affective neuroscience," Philosoph. Trans. Roy. Soc., vol. 364, no. 3, pp. 3475-3484, 2009.
51. C. Roether, L. Omlor, A. Christensen, and M. A. Giese, "Critical features for the perception of emotion from gait," J. Vis., vol. 8, 6, no. 15, pp. 1-32, 2009.
52. D. Bernhardt and P. Robinson, "Detecting affect from nonstylised body motions," in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2007, vol. 4738, pp. 59-70.
53. G. Castellano, S. Villalba, and A. Camurri, "Recognising human emotions from body movement and gesture dynamics," in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science, A. Paiva, R. Prada, and R. W. Picard, Eds. Berlin, Germany: Springer-Verlag, 2007, vol. 4738, pp. 71-82.
54. A. Kleinsmith, T. Fushimi, and N. Bianchi-Berthouze, "An incremental and interactive affective posture recognition system," in Proc. Int. Workshop Adapting Interaction Style to Affective Factors, 2005.
55. A. Kleinsmith and N. Bianchi-Berthouze, "Recognizing affective dimensions from body posture," in Affective Computing and Intelligent Interaction, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2007, vol. 4738, pp. 48-58.
56. A. Camurri, B.M.Mazzarino, R. Timmers, and G. Volpe, "Multimodal analysis of expressive gesture in music and dance performances," in Gesture-Based Communication in HCI, ser. Lecture Notes in Artificial Intelligence. Cambridge, MA: MIT Press, 2004, pp. 20-39. (Pubitemid 38285426)
57. H. Gunes and M. Piccardi, "Bi-modal emotion recognition from expressive face and body gestures," J. Netw. Comput. Appl., vol. 30, pp. 1334-1345, 2007. (Pubitemid 47263518)
58. A. Camurri, B. Mazzarini, and G. Volpe, "Analysis of expressive gesture: The eyesweb expressive gesture processing library," in Gesture-Based Communication in HCI, ser. Lecture Notes in Artificial Intelligence. Cambridge, MA: MIT Press, 2004, vol. 2915, pp. 460-467. (Pubitemid 38285465)
59. F. Pollick, H. Paterson, A. Bruderlin, and A. Sanford, "Perceiving affect from arm movement," Cognition, vol. 82, pp. 51-61, 2001.
60. A. Kleinsmith and N. Bianchi-Berthouze, "Affective body expression perception and recognition: A survey," IEEE Trans. Affective Comput., DOI: 10.1109/T-AFFC.2012.16, to be published.
61. A. Kleinsmith, N. Bianchi-Berthouze, and A. Steed, "Automatic recognition of non-acted affective postures," IEEE Trans. Syst. Man Cybern. B, Cybern., vol. 41, no. 4, pp. 1027-1038, Aug. 2011.
62. A. Kleinsmith and N. Bianchi-Berthouze, "Form as a cue in the automatic recognition of non-acted affective body expressions," in Affective Computing and Intelligent Interaction, ser. LectureNotes in Computer Science. Berlin, Germany: Springer-Verlag, 2011, vol. 6975, pp. 155-164.
63. M. Kitagawa and B. Windsor, MoCap for Artists: Workflow and Techniques for Motion Capture. Oxon, U.K.: Focal Press, 2008, pp. 190-194.
64. C. Storm and T. Storm, "A taxonomic study of the vocabulary of emotions," J. Personality Social Psychol., vol. 53, no. 4, pp. 805-816, 1987.
65. R. von Laban, Modern Educational Dance. Braintree, MA: Mac-Donald & Evans, 1963.
66. N. Savva and N. Bianchi-Berthouze, "Automatic recognition of affective body movement in a video game scenario," in Proc. Int. Conf. Intell. Technol. Interactive Entertain., A. Camurri, C. Costa, and G. Volpe, Eds., 2012, vol. 78, pp. 149-158.
67. J. L. Elman, "Finding structure in time," Cogn. Sci., vol. 14, pp. 179-211, 1990.
68. S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, 1999, pp. 754-777.
69. M. Bod́en, The DALLAS Project. Report from the NUTEK-Supported Project AIS-8: Application of Data Analysis with Learning Systems, SICS Tech. Rep. T2002:03, 1999-2001.
70. H. Meng and N. Bianchi-Berthouze, "Naturalistic affective expression classification by a multi-stage approach based on hidden Markov models," in Affective Computing and Intelligent Interfaces, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2011, vol. 6975, pp. 378-387.
71. G. Ramirez, T. Baltrusaitis, and L. P. Morency, "Modeling latent discriminative dynamic of multi-dimensional affective signals," in Affective Computing and Intelligent Interfaces, ser. Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2011, vol. 6975, pp. 396-406.
72. J. H. Riskind, "They stoop to conquer: Guiding and self-regulatory functions of physical posture after success and failure," J. Personality Social Psychol., vol. 47, no. 3, pp. 479-493, 1984.
73. C. Detels, "Whiter feminist aesthetics," Action, Criticism, Theory for Music Educ. vol. 5, no. 1, 2006 [Online]. Available: http://act.maydaygroup. org/articles/Detels5-1.pdf
74. P. Atkinson, W. H. Dittrich, A. J. Gemmell, and A. W. Young, "Evidence for distinct contributions of form and motion information to the recognition of emotions from body gestures," Cognition, vol. 104, pp. 59-72, 2007. (Pubitemid 46627560)
75. A. P. Atkinson, "Impaired recognition of emotions from body movements is associated with elevated motion coherence thresholds in autism spectrum disorders," Neuropsychologia, vol. 47, no. 13, pp. 3023-3029, 2009.