Human perception of a conversational virtual human: An empirical study on the effect of emotion and culture

Virtual Reality

Volumn 17, Issue 4, 2013, Pages 307-321

  Qu, Chao ^a   Brinkman, Willem Paul ^a   Ling, Yun ^a   Wiggers, Pascal ^a   Heynderickx, Ingrid ^a,b  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b PHILIPS RESEARCH LABORATORIES   (Netherlands)

Author keywords

Affective computing; Culture; Emotion; Virtual human; Virtual reality

Indexed keywords

AFFECTIVE COMPUTING; CULTURAL BACKGROUNDS; EMOTION; EMOTIONAL EXPRESSIONS; EMPIRICAL STUDIES; PARALLEL COMPARISONS; VIRTUAL HUMANS; VIRTUAL REALITY EXPOSURE THERAPIES;

CELL CULTURE; VIRTUAL REALITY;

BEHAVIORAL RESEARCH;

EID: 84886408264     PISSN: 13594338     EISSN: 14349957     Source Type: Journal    
DOI: 10.1007/s10055-013-0231-z     Document Type: Article

References (93)

1. Bailenson JN, Yee N, Merget D, Schroeder R (2006) The effect of behavioral realism and form realism of real-time avatar faces on verbal disclosure, nonverbal disclosure, emotion recognition, and copresence in dyadic interaction. Presence Teleoper Virtual Environ 15(4): 359-372.
2. Barrett LF (2006) Solving the emotion paradox: categorization and the experience of emotion. Pers Soc Psychol Rev Off J Soc Pers Soc Psychol 10: 20-46. doi: 10. 1207/s15327957pspr1001_2.
3. Bartneck C (2001) Affective expressions of machines. CHI '01 extended abstracts on human factors in computing systems, pp 189-190.
4. Bates J (1994) The role of emotion in believable agents. Commun ACM 37(7): 122-125.
5. Bradley MM, Lang PJ (2007) The international affective digitized sounds (2nd Edition; IADS-2): affective ratings of sounds and instruction manual. Technical report B-3. University of Florida, Gainesville, Fl.
6. Brand M (1999) Voice puppetry. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques SIGGRAPH 99, pp 21-28.
7. Bregler C (1997) Video rewrite: driving visual speech with audio. In: Proceedings of SIGGRAPH'97, pp 1-8.
8. Brinkman W-P, Veling W, Dorrestijn E, Sandino G, Vakili V, van der Gaag M (2011) Virtual reality to study responses to social environmental stressors in individuals with and without psychosis. Stud Health Technol Inf 167: 86-91.
9. Brinkman W-P, Hartanto D, Kang N, de Vliegher D, Kampmann IL, Morina N et al (2012) A virtual reality dialogue system for the treatment of social phobia. In: Paper presented at the CHI'12 extended abstracts on human factors in computing systems.
10. Broekens J, Brinkman W-P (2009) Affectbutton: towards a standard for dynamic affective user feedback. Affect Comput Intel Interact Workshops 2009: 1-8.
11. Broekens J, Brinkman W-P (2013) AffectButton: a method for reliable and valid affective self-report. Int J Hum Comput Stud 71(6): 641-667.
12. Broekens J, Pronker A, Neuteboom M (2010) Real time labeling of affect in music using the affectbutton. In: Proceedings of the 3rd international workshop on affective interaction in natural environments, pp 21-26.
13. Broekens J, Harbers M, Brinkman W-P, Jonker C, van den Bosch K, Meyer JJ (2011) Validity of a virtual negotiation training. In: IVA'11 Proceedings of the 11th international conference on intelligent virtual agents, pp 435-436.
14. Broekens J, Harbers M, Brinkman W-P, Jonker C, van den Bosch K, Meyer J-J (2012) Virtual reality negotiation training increases negotiation knowledge and skill. In: IVA'12 Proceedings of the 12th international conference on intelligent virtual agents, pp 218-230.
15. Broekens J, Qu C, Brinkman W-P (2012) Dynamic facial expression of emotion made easy. Technical report. Interactive Intelligence, Delft University of Technology, pp 1-30.
16. Cassell J, Thorisson KR (1999) The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents. Appl Artif Intell 13: 519-538.
17. Cassell J, Pelachaud C, Badler N, Steedman M, Achorn B, Becket T et al (1994) Animated conversation: Rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In: Proceedings of ACM SIGGRAPH, pp 413-420.
18. Cerezo E, Baldassarri S (2008) Affective embodied conversational agents for natural interaction. In: Or J (ed) Affective computing: emotion modelling, synthesis and recognition, pp 329-354.
19. Chuang E, Bregler C (2002) Performance driven facial animation using blendshape interpolation. Computer Science Technical Report, Stanford University.
20. Church T, Katigbak M (1998) Language and organisation of Filipino emotion concepts: comparing emotion concepts and dimensions across cultures. Cogn Emot 12(1): 63-92.
21. Cohen MM, Massaro DW (1993) Modeling coarticulation in synthetic visual speech. In: Thalman NM, Thalman D (eds) Models and Techniques in Computer Animation. Springer, Verlag, pp 139-156.
22. Core M, Traum D, Lane HC, Swartout W, Marsella S, Gratch J, van Lent M (2006) Teaching negotiation skills through practice and reflection with virtual humans. Simulation 82: 685-701.
23. Cowell AJ, Stanney KM (2003) Embodiment and interaction guidelines for designing credible, trustworthy embodied conversational agents. In: 4th international workshop on intelligent virtual agents IVA 2003, 2792, pp 301-309.
24. Darwin C (1872) The expression of emotion in man and animals. Philosophical Library, New York.
25. Dotsch R, Wigboldus DHJ (2008) Virtual prejudice. J Exp Soc Psychol 44(4): 1194-1198.
26. Ekman P (1994) Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. Psychological Bull 115(2): 268-287.
27. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc 17(2): 124-129.
28. Ekman P, Friesen WV, O'Sullivan M, Chan A, Diacoyanni-Tarlatzis I, Heider K, Ricci-Bitti P (1987) Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol 53(4): 712-717.
29. Ekman P, Rolls ET, Perrett DI, Ellis HD (1992) Facial expressions of emotion: an old controversy and new findings. Philos Trans Biol Sci 335: 63-69.
30. Ekman P, Friesen WV, Hager JC (2002) Facial action coding system. Human Face 97: 4-5.
31. Elfenbein H (2003) Universals and cultural differences in recognizing emotions. Curr Dir Psychol Sci 12(5): 159-164.
32. Elfenbein HA, Ambady N (2002) Is there an in-group advantage in emotion recognition? Psychol Bull 128(2): 243-249.
33. Elfenbein HA, Beaupre M, Levesque M, Hess U (2007) Toward a dialect theory: cultural differences in the expression and recognition of posed facial expressions. Emotion (Washington, DC) 7(1): 131-146.
34. Endrass B, Rehm M, Lipi A (2011) Culture-related differences in aspects of behavior for virtual characters across Germany and Japan. In: Proceedings of AAMAS'11, 2, pp 441-448.
35. Ersotelos N, Dong F (2008) Building highly realistic facial modeling and animation: a survey. Visual Comput 24(1): 13-30.
36. Ezzat T, Geiger G, Poggio T (2004) Trainable videorealistic speech animation. In: Sixth IEEE international conference on automatic face and gesture recognition 2004 proceedings, pp 57-64.
37. Fox E (2008) Emotion science cognitive and neuroscientific approaches to understanding human emotions. Palgrave Macmillan, UK.
38. Gratch J, Rickel J, Andre E, Cassell J, Petajan E, Badler NI, Jeff R (2002) Creating interactive virtual humans: some assembly required. IEEE Intell Syst 17(4): 54-63.
39. Gross MM, Crane EA, Fredrickson BL (2010) Methodology for assessing bodily expression of emotion. J Nonverbal Behav 34: 223-248. doi: 10. 1007/s10919-010-0094-x.
40. Haring M, Bee N, Andre E (2011) Creation and evaluation of emotion expression with body movement, sound and eye color for humanoid robots. In: RO-MAN, 2011 IEEE, pp 204-209.
41. Hofstede G (2001) Culture's consequences: comparing values, behaviors, institutions and organisations across nations. Sage Publications, Thousand Oaks.
42. Hudlicka E, Delft TU (2009) Foundations for modelling emotions in game characters: modelling emotion effects on cognition. In: Affective computing and intelligent interaction and workshops, ACII 2009.
43. Irtel H (2007) PXLab: the psychological experiments laboratory [online]. University of Mannheim, Mannheim.
44. Isbister K (2006) Better game characters by design: a psychological approach. Education, CRC Press.
45. Jack RE, Garrod OGB, Yu H, Caldara R, Schyns PG (2012) Facial expressions of emotion are not culturally universal. Proc Natl Acad Sci 109(19): 7241-7244.
46. Jan D, Herrera D, Martinovski B (2007) A computational model of culture-specific conversational behavior. In: IVA '07 Proceedings of the 7th international conference on intelligent virtual agents, pp 45-56.
47. Kahler K, Haber J, Seidel H-P (2001) Geometry-based muscle modeling for facial animation. In: Proceedings of graphics interface, pp 37-46.
48. Keltner D, Ekman P (2000) Facial expression of emotion. Handbook of emotions, 2nd edn. pp 236-249.
49. Kleinsmith A, Bianchi-Berthouze N (2013) Affective body expression perception and recognition: a survey. IEEE Trans Affect Comput 4: 15-33. doi: 10. 1109/T-AFFC. 2012. 16.
50. Kleinsmith A, De Silva PR, Bianchi-Berthouze N (2006) Cross-cultural differences in recognizing affect from body posture. Interact Comput 18(6): 1371-1389.
51. Kulms P, Kramer NC, Gratch J, Kang S-H (2011) It's in their eyes: a study on female and male virtual humans' gaze. In: IVA'11 Proceedings of the 11th international conference on intelligent virtual agents, pp 80-92.
52. Lance BJ, Rey MD, Marsella SC (2008) A model of gaze for the purpose of emotional expression in virtual embodied agents. In: AAMAS '08 proceedings of the 7th international joint conference on autonomous agents and multiagent systems, 1, pp 12-16.
53. Lang PJ (1995) The emotion probe. Studies of motivation and attention. Am Psychol 50(5): 372-385.
54. Lang PJ, Bradley MM, Cuthbert BN (1999) International affective picture system (IAPS): technical manual and affective ratings. Psychology. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL.
55. Lang PJ, Bradley MM, Cuthbert BN (2008) International affective picture system (IAPS): affective ratings of pictures and instruction manual. Technical Report A-8.
56. Lee J, Marsella SC (2012) Modeling speaker behavior: a comparison of two approaches. In: IVA'12 Proceedings of the 12th international conference on intelligent virtual agents, pp 161-174.
57. Lee Y, Terzopoulos D, Walters K (1995) Realistic modeling for facial animation. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques SIGGRAPH 95, pp 55-62.
58. Lee J, Prendinger H, Neviarouskaya A, Marsella S (2009) Learning models of speaker head nods with affective information. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops, pp 1-6.
59. Link M, Armsby P, Hubal RC, Guinn CI (2006) Accessibility and acceptance of responsive virtual human technology as a survey interviewer training tool. Comput Hum Behav 22: 412-426. doi: 10. 1016/j. chb. 2004. 09. 008.
60. Litwinowicz P, Williams L (1994) Animating images with drawings. In: Proceedings of the 21st annual conference on computer graphics and interactive techniques SIGGRAPH 94, pp 409-412.
61. MacDorman KFK, Coram JJA, Ho C-CC, Patel H (2010) Gender differences in the impact of presentational factors in human character animation on decisions in ethical dilemmas. Presence Teleoper Virtual Environ 19(3): 213-229.
62. Matsumoto D (2002) Methodological requirements to test a possible in-group advantage in judging emotions across cultures: comment on Elfenbein and Ambady (2002) and evidence. Psychol Bull 128(2): 236-242.
63. Matsumoto D (2007) Emotion judgments do not differ as a function of perceived nationality. Int J Psychol 42(3): 207-214.
64. Mehrabian A, Russell JA (1974) An approach to environmental psychology. MIT Press, Cambridge, MA.
65. Melo Cd, Carnevale P, Gratch J (2011) The effect of expression of anger and happiness in computer agents on negotiations with humans. In: The tenth international conference on autonomous agents and multiagent systems, pp 2-6.
66. Moridis CN, Economides AA (2012) Affective learning: empathetic agents with emotional facial and tone of voice expressions. IEEE Trans Affect Comput 3(3): 260-272.
67. Morris JD (1995) Observations: SAM the self-assessment manikin an efficient cross-cultural measurement of emotional response. J Advert Res 35(6): 63-68.
68. Ochsman RB, Chapanis A (1974) The effects of 10 communication modes on the behavior of teams during co-operative problem-solving. Int J ManMachine Stud 6(5): 579-619.
69. Opris D, Pintea S, Garcia-Palacios A, Botella CM, Szamoskozi S, David D (2012) Virtual reality exposure therapy in anxiety disorders: a quantitative meta-analysis. Depression Anxiety 29: 85-93. doi: 10. 1002/da. 20910.
70. Parke FI (1972) Computer generated animation of faces. Proc ACM Annu Conf 1: 451-457.
71. Parke FI (1974) A parametric model for human faces. The University of Utah, Doctoral Dissertation.
72. Petrushin V (1999) Emotion in speech: recognition and application to call centers. In: Artificial neural network in engineering (ANNIE'99), pp 7-10.
73. Picard RW (1998) Toward agents that recognize emotion. In: Actes proceedings IMAGINA, pp 153-165.
74. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10): 1175-1191.
75. Platt SM, Badler NI (1981) Animating facial expressions. ACM SIGGRAPH Comput Graph 15(3): 245-252.
76. Qiu L, Benbasat I (2005) Online consumer trust and live help interfaces: the effects of text-to-speech voice and three-dimensional avatars. Int J Human-Comput Interact 19: 37-41.
77. Reeves B, Nass C (1996) The media equation. Cambridge University Press, Cambridge.
78. Rizzo A, Lange B, Buckwalter J, Forbell E, Kim J, Sagae K, Kenny P (2011) An intelligent virtual human system for providing healthcare information and support. Stud Health Technol Inf 163: 503-509.
79. Russell JA (1991) Culture and the categorization of emotions. Psychol Bull 110: 426-450.
80. Ruttkay Z, Pelachaud C (2005) From brows to trust: evaluating embodied conversational agents. Springer, Berlin.
81. Scherer KR (1995) Expression of emotion in voice and music. J Voice Off J Voice Found 9(3): 235-248.
82. Scherer KR (2003) Vocal communication of emotion: a review of research paradigms. Speech Commun 40: 227-256.
83. Schiano DJ, Ehrlich SM, Rahardja K, Sheridan K (2000) Face to interface: facial affect in (hu)man and machine. Proc ACM CHI 2000: 193-200.
84. Schroder M (2004) Speech and emotion research: an overview of research frameworks and a dimensional approach to emotional speech synthesis. Research Report of the Institute of Phonetics.
85. Slater M, Pertaub D-P, Steed A (1999) Public speaking in virtual reality: facing an audience of avatars. IEEE Comput Graphics Appl 19(2): 6-9.
86. Tomkins SS (1962) Affect, imagery, consciousness: vol 1. The positive affects. Springer, New York.
87. Tomkins SS (1963) Affect, imagery, consciousness: vol 2. The negative affects. Springer, New York.
88. Tsapatsoulis N, Raouzaiou A, Kollias S, Cowie R, Douglas-Cowie E (2002) Emotion recognition and synthesis based on MPEG-4 FAPs. In; MPEG-4 facial animation the standard implementations applications.
89. Waters K (1987) A muscle model for animating three-dimensional facial expression. Comput Graph SIGGRAPH Proc 21(4): 17-24.
90. Wierzbicka A (1995) Emotions across languages and cultures: diversity and universals. Cambridge University Press, Cambridge.
91. Wong JW-E, McGee K (2012) Frown more, talk more: effects of facial expressions in establishing conversational rapport with virtual agents. In: IVA'12 Proceedings of the 12th international conference on intelligent virtual agents, pp 419-425.
92. Yun C, Deng Z, Hiscock M (2009) Can local avatars satisfy a global audience? A case study of high-fidelity 3D facial avatar animation in subject identification and emotion perception by US and international groups. Comput Entertain 7(2): 1-25.
93. Zhang QZQ, Liu Z, Quo GQG, Terzopoulos D, Shum H-YSH-Y (2006) Geometry-driven photorealistic facial expression synthesis. IEEE Trans Visual Comput Graphics 12(1): 48-60.