Modelling multimodal expression of emotion in a virtual agent

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 364, Issue 1535, 2009, Pages 3539-3548

  Pelachaud, Catherine ^a  

^a INSTITUT TELECOM   (France)

Author keywords

Embodied conversational agent; Emotion; Non verbal behaviour

Indexed keywords

NUMERICAL MODEL; PSYCHOLOGY; SIGNAL; TEMPORAL EVOLUTION; TEMPORAL VARIATION;

EID: 73549112229     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2009.0186     Document Type: Article

References (75)

1. Albrecht, I., Schroeder, M., Haber, J. & Seidel, H.-P. 2005 Mixed feelings: expression of non-basic emotions in a muscle-based talking head. Virtual Real, (special issue) 8, 201-212. (doi:10.1007/s10055-005-0153-5). (Pubitemid 41386797)
2. André, E., Rehm, M., Minker, W. & Buhler, D. 2004 Endowing spoken language dialogue systems with emotional intelligence. In Affective dialogue systems (eds E. André, L. Dybkjaer, W. Minker & P. Heisterkamp), pp. 178-187. Berlin/Heidelberg, Germany, Springer Verlag. (Pubitemid 38851679)
3. Arya, A., DiPaola, S. & Parush, A. 2009 Perceptually valid facial expressions for character-based applications. Int. J. Comput. Game Technol. Article ID 462315. (doi: 10.1155/2009/462315)
4. Austin, J. L. 1962 How to do thinks with words. London, UK: Oxford University Press.
5. Ballin, D., Gillies, M. F. & Crabtree, I. B. 2004 A framework for interpersonal attitude and non-verbal communication in improvisational visual media production. In First European Conf. on Visual Media Production (CVMP), pp. 203-210, London, UK. (Pubitemid 40954849)
6. Bänziger, T. & Scherer, K. 2007 Using actor portrayals to systematically study multimodal emotion expression: the GEMEP corpus. In Affective computing and intelligent interaction. Lecture Notes in Computer Science, no. 4738/ 2007. Berlin/Heidelberg: Springer.
7. Becker, C. & Wachsmuth, I. 2006 Modeling primary and secondary emotions for a believable communication agent. In Int. Workshop on Emotion and Computing, in conj. with the 29th annual German Conf. on Artificial Intelligenz (KI2006), pp. 31-34, Bremen, Germany.
8. Beneveniste, E. 1974 Problèmes de linguistique générak. Paris, France: Gallimard.
9. Beskow, J. Animation of talking agents. 1997 In Proc. ESCA Workshop on Audio-Visual Speech Processing (eds C. Benoit & R. Campbell), pp. 149-152. Rhodes, Greece.
10. Beskow, J., Granstrom, B. & House, D. 2006 Visual correlates to prominence in several expressive modes. In Proceedings of Interspeech, pp. 1272-1275, Pittsburg, PA.
11. Bevacqua, E., Mancini, M., Niewiadomski, R. & Pelachaud, C. 2007 An expressive ECA showing complex emotions. In AISB'07 Annual convention, workshop "Language, Speech and Gesture for Expressive Characters", pp. 208-216, Newcastle UK.
12. Bolinger, D. 1989 Intonation and its uses. Stanford, CA: Stanford University Press.
13. Buisine, S., Abrilian, S., Niewiadomski, R., Martin, J.-C., Devillers, L. & Pelachaud, C. 2006 Perception of blended emotions: from video corpus to expressive agent. In The 6th International Conference on Intelligent Virtual Agents, Marina del Rey, USA.
14. Caridakis, G., Raouzaiou, A., Bevacqua, E., Mancini, M., Karpouzis, K., Malatesta, L. & Pelachaud, C. 2008 Virtual agent multimodal mimicry of humans. In Language resources and evaluation, special issue on Multimodal Corpora for Modelling Human Multimodal Behavior, vol.41 (eds J.-C. Martin, P. Paggio, P. K'uhnlein, R. Stiefelhagen & F. Pianesi), pp. 367-388. Berlin, Heidelberg: Springer.
15. Cassell, J., Pelachaud, C., Badler, N. L., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S. & Stone, M. 1994 Animated conversation: rule-based generation of facial expression, gesture and spoken intonation for multiple conversational agents. In Computer Graphics Proc. Annual Conf. Series, p. 413-420. ACM SIGGRAPH.
16. Castelfranchi, C. & Parisi, D. 1980 Linguaggio, conoscenze e scopi. Bologna, Italy: II Mulino.
17. Castellano, G., Villalba, S. D. & Camurri, A. 2007 Recognising human emotions from body movement and gesture dynamics. In ACII, pp. 71-82.
18. Cervoni, J. 1987 Lénonciation. Linguistique nouvelle. Paris, France: Presses Universitaires de France.
19. Chi, D. M., Costa, M., Zhao, L. & Badler, N. I. 2000 The EMOTE model for effort and shape. In Siggraph 2000, Computer Graphics Proceedings (ed. K. Akeley), pp. 173-182. Boston, MA: ACM Press/ACM SIGGRAPH/ Addison Wesley Longman.
20. DeCarolis, B., Pelachaud, C., Poggi, I. & de Rosis, F. 2001 Behavior planning for a reflexive agent. In IJCAI'01, Seattle, USA.
21. DeCarolis, B., Pelachaud, C., Poggi, I. & Steedman, M. 2004 APML, a mark-up language for believable behavior generation. In Life-like characters. Tools, affective functions and applications (eds H. Prendinger & M. Ishizuka), pp. 65-85. Berlin/Heidelberg: Springer.
22. Devillers, L., Abrilian, S. & Martin, J.-C. 2005 Representing real life emotions in audiovisual data with non basic emotional patterns and context features. In 1st Int. Conf. Affective Computing & Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October 2005.
23. Ducrot, O. 1984 Le dire et le dit. Paris, France: Les Editions de Minuit.
24. Ducrot, O. et al. 1980 Les mots du discours. Paris, France: Les Editions de Minuit.
25. Duy Bui, Heylen, D., Poel, M. & Nijholt, A. 2004 Combination of facial movements on a 3D talking head. In Computer Graphics International, pp. 284-291.
26. Ekman, P. 1989 The argument and evidence about universals in facial expressions of emotion. In Handbook of social psychophysiology (eds H. Wagner & A. Manstead), pp. 143-164. Chichester, UK: Wiley.
27. Ekman, P. 2003 Emotions revealed. New York/London: Times Books (US)/Weidenfeld & Nicolson.
28. Ekman, P. & Friesen, W. 1975 Unmasking the face: a guide to recognizing emotions from facial clues. Eaglewood Cliffs, NJ: Prentice-Hall, Inc.
29. Ekman, P., Friesen, W. & Hager, J. 2002 Facial action coding system: the manual & user' guide. Salt Lake City, UT: A Human Face.
30. Garcia-Rojas, A., Vexo, F., Thalmann, D., Raouzaiou, A., Karpouzis, K., Kollias, S., Moccozet, L. & MagnenatThalmann, N. 2006 Emotional face expression profiles supported by virtual human ontology. Comp. Anim. Virtual Worlds 17, 259-269. (doi:10.1002/cav.130). (Pubitemid 44031457)
31. Grammer, K. & Oberzaucher, E. 2006 The reconstruction of facial expressions in embodied systems. ZIF Mitteiluagen 2/2006, pp. 14-31.
32. Hartmann, B., Mancini, M. & Pelachaud, C. 2005 Implementing expressive gesture synthesis for embodied conversational agents. In Gesture in Human-Computer lnteraction and Simulation, 6th International Gesture Workshop, GW2005, Berder Island, pp. 188-199.
33. Johnson, W. L., Rizzo, P., Bosma, W., Kole, S., Ghijsen, M. & van Welbergen, H. 2004 Generating socially appropriate tutorial dialog. In ISCA Workshop on Affective Dialogue Systems, pp. 254-264. (Pubitemid 38851689)
34. Keltner, D. 1995 Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame. J. Pers. Soc. Psychol. 68, 441-454. (doi: 10.1037/0022-3514.68.3.441).
35. Krahmer, E. & Swerts, M. 2004 More about brows. In From brows till trust: evaluating embodied conversational agents (eds Z. Ruttkay & C. Pelachaud) pp. 191-216. Norwell, MA: Kluwer Academic Publishers.
36. Laban, R. & Lawrence, F. C. 1974 Effort: economy in body movement. Boston, MA: Plays, Inc.
37. Liu, K. & Hertzmann, A. 2005 Learning physics-based motion style with inverse optimization. In Computer Graphics Proc. Annual Conf. Series. ACM SIGGRAPH.
38. Malatesta, L., Raouzaiou, A., Karpouzis, K. & Kollias, S. 2006 MPEG-4 facial expression synthesis based on appraisal theory. In 3rd IFIP Conf. in Artificial Intelligence Applications and Innovations.
39. Mancini, M. & Pelachaud, C. 2008 Distinctiveness in multimodal behaviors. In Seventh International Joint Conf. on Autonomous Agents and Multi-Agent Systems, AAMAS'08, Estoril, Portugal.
40. Martin, J.-C, Niewiadomski, R., Devillers, L., Buisine, S. & Pelachaud, C. 2006 Multimodal complex emotions: gesture expressivity and blended facial expressions. Int. J. HR. (special issue) 20, 477-498.
41. Ménardais, S., Kulpa, R., Multon, F. & Arnaldi, B. 2004 Synchronization for dynamic blending of motions. In Proc. of ACM, SIGGRAPHIEUROGRAPHICS. Symposium of Computer Animation, pp. 325-335, France.
42. Neff, M. & Fiume, E. 2004 Methods for exploring expressive stance. In Proc. ACM, SIGGRAPH/EUROGRAPHICS Symposium of Computer Animation, pp. 49-58, Grenoble, France.
43. Niewiadomski, R. & Pelachaud, C. 2007a Fuzzy similarity of facial expressions of embodied agents. In Proc. of the 7th Int. Conf. on Intelligent Virtual Agents (IVA) (eds C. Pelachaud, J.-C. Martin, E. André, G. Chollet, K. Karpouzis & D. Pélé), pp. 86-98. Berlin, Heidelberg: Springer Verlag. (Pubitemid 350271477)
44. Niewiadomski, R. & Pelachaud, C. 2007b Model of facial expressions management for an embodied conversational agent. In 2nd Int. Conf. on Affective Computing and Intelligent Interaction (ACII), pp. 12-23, Lisbon, Portugal.
45. Niewiadomski, R., Ochs, M. & Pelachaud, C. 2008 Expressions of empathy in ECAs. In Intelligent Virtual Agents, IVA'08, Tokyo, Japan, pp. 37-44.
46. Niewiadomski, R., Hyniewska, S. & Pelachaud, C. In press a Evaluation of multimodal sequential expressions of emotions in ECA. In Proc. Int. Conf. on Affective Computing and Intelligent lnteraction (ACII), Amsterdam, The Netherlands. Berlin/Heidelberg: Springer Verlag.
47. Niewiadomski, N., Mancini, M., Hyniewska, S. & Pelachaud, C. In press b. Emotional behaviours in embodied conversational agents. In A blueprint for an affectively competent agent (eds E. Roesch, K. R. Scherer & T. Bänziger).
48. Paleari, M. & Lisetti, C. L. 2006 Psychologically grounded avatars expressions. In First Workshop on Emotion and Computing at KI 2006, 29th Annual Conf. on Artificial Intelligence, Bremen, Germany.
49. Pandzic, I. S. & Forcheimer, R. (eds) 2002 MPEG4 facial animation - The standard, implementations and applications. Chichester, UK: John Wiley, Sons.
50. Pelachaud, C. 2005 Multimodal expressive embodied conversational agent. In ACM Multimedia, Brave New Topics session, Singapore.
51. Pelachaud, C. & Prevost, S. 1994 Sight and sound: generating facial expressions and spoken intonation from context. In Proc. of the 2nd ESCA/AAAI/IEEE workshop on Speech Synthesis, pp. 216-219, New Paltz, NY.
52. Pelachaud, C. & Poggi, I. 2002 Subtleties of facial expressions in embodied agents. J. Vis. Comput. Animat. 13, 301-312. (doi:10.1002/vis.299). (Pubitemid 135714461)
53. Pelachaud, C., Badler, N. I. & Steedman, M. 1996 Generating facial expressions for speech. Cognit. Sci. 20, 1-46.
54. Plutchik, R. 1980 Emotions: a psychoevolutionary synthesis. Harper & Row, New York.
55. Poggi, I. 2007 Mind, hands, face and body. A goal and belief view of multimodal communication, volume Körper, Zeichen, Kultur. Berlin: Weidler Verlag.
56. Poggi, I. & Pelachaud, C. 1999 Emotional meaning and expression in performative faces. In Int. Workshop on Affect in Interactions: towards a New Generation of Interfaces, Annual Conf. AC'99 of the EC 13 Programme, Siena, Italy, October 1999.
57. Poggi, I. & Pelachaud, C. 2000a Emotional meaning and expression in animated faces. In Affect in interactions (ed. A. Paiva). Berlin, Germany: Springer-Verlag.
58. Poggi, I. & Pelachaud, C. 2000b Facial performative in a conversational system. In Embodied conversational characters (eds S. Prevost, J. Cassell, J. Sullivan & E. Churchill), pp. 155-188. Cambridge, MA: MIT press.
59. Prendinger, H. & Ishizuka, M. 2001 Social role awareness in animated agents. In Proc. of the 5th Int. Conf. on Autonomous Agents, Montreal, Canada, May-June 2001.
60. Rehm, M. & André, E. 2005a Informing the design of embodied conversational agents by analysing multimodal politeness behaviors in human-human communication. In Workshop on Conversational Informatics for Supporting Social Intelligence and Interaction, 2005.
61. Rehm, M. & André, E. 2005b Catch me if you can: exploring lying agents in social settings. In AAMAS, pp. 937-944, 2005.
62. Ruttkay, Z., Noot, H. & ten Hagen, P. 2003a Emotion disc and emotion squares: tools to explore the facial expression face. Comput Graph. Forum 22, 49-53.
63. Ruttkay, Z., van Moppes, V. & Noot, H. 2003b The jovial, the reserved and the robot. In Proceedings of the AAMAS03 Ws on Embodied Conversational Characters as Individuals, Melbourne, Australia, 2003.
64. Scherer, K. 1988 Facets of emotion: recent research. Hillsdale, NJ: Lawrence Erlbaum Associates Publishers.
65. Scherer, K. R. 2001 Appraisal considered as a process of multilevel sequential checking. In Appraisal processes in emotion: theory, methods, research (eds K. Scherer, A. Schorr & T. Johnstone), pp. 92-119. New York, NY: Oxford University Press.
66. Scherer, K. R. & Ellgring, H. 2007 Multimodal expression of emotion: affect programs or componential appraisal patterns? Emotion 7, 158-171. (doi:10.1037/1528-3542. 7.1.158). (Pubitemid 46433172)
67. Stoiber, N., Séguier, R. & Breton, G. 2009 Automatic design of a control interface for a synthetic face. In IUI, pp. 207-216, 2009.
68. Steedman, M. 1991 Structure and intonation. Language 67, 260-296. (doi:10.2307/415107).
69. Stone, M. & DeCarlo, D. 2003 Crafting the illusion of meaning: template-based generation of embodied conversational behavior. In Computer Animation and Social Agents, pp. 11-16. Brunswick, NJ.
70. Swerts, M. & Krahmer, E. 2008. Facial expressions and prosodic prominence: comparing modalities and facial areas. J. Phon. 36,219-238. (doi:10.1016/j.wocn.2007.05.001).
71. Torres, O., Cassell, J. & Prevost, S. 1997 Modeling gaze behavior as a function of discourse structure. In Machine conversation (ed Y Wilks), First International Workshop on Human-Computer Conversations, Bellagio, Italy. Kluwer.
72. Tsapatsoulis, N., Raouzaiou, A., Kollias, S., Cowie, R. & Douglas-Cowie, E. 2002 Emotion recognition and synthesis based on MPEG-4 FAPs in MPEG-4 facial animation. In MPEG4 facial animation - the standard, implementations and applications (eds I. S. Pandzic & R. Forcheimer). Chichester, UK: John Wiley & Sons.
73. Wallbott, H. 1998 Bodily expression of emotion. Eur. J. Soc. Psychol. 28, 879-896. (doi: 10.1002/(SICI) 1099-0992(1998110)28:6<879::AID-EJSP901>3. 0.CO;2-W)
74. Wallbott, H. G. & Scherer, K. 1986 Cues and channels in emotion recognition. J. Pers. Soc. Psychol. 24.
75. Whissell, C. 1989 The dictionary of affect in language. In Emotion: Theory, Research, and Experience, volume 4: The Measurement of Emotions (eds R. Plutchik & H. Kellerman), pp. 113-131. San Diego, CA: Academic Press, Inc.