Multimodal complex emotions: Gesture expressivity and blended facial expressions

International Journal of Humanoid Robotics

Volumn 3, Issue 3, 2006, Pages 269-291

  Martin, Jean Claude ^a   Niewiadomski, Radoslaw ^b   Devillers, Laurence ^a   Buisine, Stephanie ^c   Pelachaud, Catherine ^d  

^a CNRS   (France)

^b UNIVERSITY OF PERUGIA   (Italy)

^c Laboratoire Conception de Produits et Innovation   (France)

^d UNIVERSITÉ PARIS 8   (France)

Author keywords

Corpus; Embodied conversational agent; Emotion; Multimodality

Indexed keywords

COMPUTATIONAL COMPLEXITY; FACE RECOGNITION; GESTURE RECOGNITION; MATHEMATICAL MODELS;

CORPUS; EMBODIED CONVERSATIONAL AGENT; MULTIMODAL COMPLEX EMOTIONS;

INTELLIGENT AGENTS;

EID: 33750201407     PISSN: 02198436     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0219843606000825     Document Type: Article

References (62)

1. P. Ekman and W. V. Friesen, Unmasking the Face. A Guide to Recognizing Emotions from Facial Clues (Prentice-Hall Inc., Englewood Cliffs, NJ, 1975).
2. M. Schröder, Experimental study of affect burst, Speech Commun. 40(1-2) (2003) 99-116 [Special Issue following the ISCA Workshop on Speech and Emotion].
3. R. Cowie, Emotional states expressed in speech, in ISCA ITRW on Speech and Emotion: Developing a Conceptual Framework for Research, Newcastle, Northern Ireland (5-7 September, 2000), pp. 224-231.
4. R. Banse and K. Scherer, Acoustic profiles in vocal emotion expression, J. Pers. Soc. Psychol. 70(3) (1996) 614-636.
5. M. DeMeijer, The contribution of general features of body movement to the attribution of emotions, J. Nonverbal Behav. 13 (1989) 247-268.
6. J. Newlove, Laban for Actors and Dancers (Routledge, New York, 1993).
7. R. T. Boone and J. G. Cunningham, Children's decoding of emotion in expressive body movement: The development of cue attunement, Dev. Psychol. 34(5) (1998) 1007-1016.
8. H. G. Wallbott, Bodily expression of emotion, Eur. J. Soc. Psychol. 28 (1998) 879-896.
9. A. Kapur, A. Kapur, N. Virji-Babul, G. Tzanetakis and P. F. Driessen, Gesture-based affective computing on motion capture data, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), 22-24 October (Springer, Beijing, China, 2005), pp. 1-8.
10. K. R. Scherer, Analyzing emotion blends, in Proc. 10th Conf. Int. Soc. for Research on Emotions (Fischer, A., Würzburg, Germany, 1998), pp. 142-148.
11. L. Devillers, L. Vidrascu and L. Lamel, Emotion detection in real-life spoken dialogs recorded in call center, J. Neural Network 18(4) (2005) 407-422 [Special issue: Emotion and Brain].
12. V. P. Richmond and J. C. Croskey, Non Verbal Behavior in Interpersonal Relations (Allyn & Bacon Inc., 1999).
13. P. Ekman and W. Friesen, Felt, false, miserable smiles, J. Nonverbal Behav. 6(4) (1982) 238-251.
14. M. Wiggers, Judgments of facial expressions of emotion predicted from facial behavior, J. Nonverbal Behav. 7(2) (1982) 101-116.
15. I. S. Pandzic and R. Forchheimer, MPEG-4 Facial Animation. The Standard, Implementation and Applications (John Wiley & Sons, Ltd., 2002).
16. M. Rehm and E. André, Catch me if you can - Exploring lying agents in social settings, Int. Conf. Autonomous Agents and Multiagent Systems (AAMAS'2005), Utrecht, The Netherlands, 25-29 July (ACM, 2005), pp. 937-944.
17. J. N. Bassili, Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face, J. Pers. Soc. Psychol. 37(11) (1979) 2049-2058.
18. K. Gouta and M. Miyamoto, Emotion recognition, facial components associated with various emotions, Shinrigaku Kenkyu 71(3) (2000) 211-218.
19. E. Constantini, F. Pianesi and M. Prete, Recognizing emotions in human and synthetic faces: The role of the upper and lower parts of the face, in Intelligent User Interfaces (IUI'05), San Diego, CA, USA, 9-12 January (ACM, 2005), pp. 20-27.
20. J. T. Cacioppo, R. P. Petty, M. E. Losch and H. S. Kim, Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions, J. Pers. Soc. Psychol. 50 (1986) 260-268.
21. J. Ostermann, Animation of synthetic faces in MPEG-4, in Computer Animation'98, Philadelphia, USA (8-10 June, 1998), pp. 49-51.
22. I. Albrecht, M. Schröder, J. Haber and H.-P. Seidel, Mixed feelings: Expression of nonbasic emotions in a muscle-based talking head, J. Virtual Reality 8(4) (2005) 201-212 [Special issue on "Language, Speech & Gesture"].
23. N. Tsapatsoulis, A. Raouzaiou, S. Kollias, R. Cowie and E. Douglas-Cowie, Emotion recognition and synthesis based on MPEG-4 FAPs, in MPEG-4 Facial Animation, eds. I. S. Pandzic and R. Forchheimer (John Wiley & Sons, UK, 2002), pp. 141-167.
24. Z. Ruttkay, H. Noot and P. ten Hagen, Emotion disc and emotion squares: Tools to explore the facial expression face, Comput. Graph. Forum 22(1) (2003) 49-53.
25. T. D. Bui, Creating emotions and facial expressions for embodied agents, thesis, Department of Computer Science (Taalwitgeverij Neslia Paniculata, Enschede, 2004).
26. P. R. De Silva, A. Kleinsmith and N. Bianchi-Berthouze, Towards unsupervised detection of affective body posture nuances, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October (Springer, 2005), pp. 32-40.
27. H. Gunes and M. Piccardi, Fusing face and body display for bi-modal emotion recognition: Single frame analysis and multi-frame post integration, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October (Springer, 2005), pp. 102-110.
28. R. el Kaliouby and P. Robinson, Generalization of a vision-based computational model of mind-reading, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October (Springer, 2005), pp. 582-590.
29. S. M. Choi and Y. G. Kim, An affective user interface based on facial expression recognition and eye-gaze tracking, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October (Springer, 2005), pp. 907-915.
30. E. Douglas-Cowie, N. Campbell, R. Cowie and P. Roach, Emotional speech: Towards a new generation of databases, Speech Commun. 40 (2003) 33-60.
31. Y. Cao, P. Faloutsos, E. Kohler and F. Pighin, Real-time speech motion synthesis from recorded motions, in ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Grenoble, France (27-29 August, 2004), pp. 347-355.
32. A. Egges, S. Kshirsagar and N. Magnenat-Thalmann, Imparting individuality to virtual humans, in Int. Workshop Virtual Reality Rehabilitationy, Lausanne, Switzerland (7-8 November, 2002), pp. 201-108.
33. I. S. Pandzic, Facial motion cloning, Graph. Models 65(6) (2003) 385-404.
34. M. Kipp, Gesture Generation by Imitation. From Human Behavior to Computer Character Animation (Boca Raton, Dissertation.com, Florida, 2004).
35. J. Cassell, M. Stone and Y. Hao, Coordination and context-dependence in the generation of embodied conversation, in 1st Int. Natural Language Generation Conf. (INLG'2000), Mitzpe Ramon, Israel (12-16 June, 2000), pp. 171-178.
36. S. Abrilian, L. Devillers, S. Buisine and J.-C. Martin, EmoTV1: Annotation of reallife emotions for the specification of multimodal affective interfaces, in 11th Int. Conf. Human-Computer Interaction (HCII'2005), LEA, Las Vegas, Nevada, USA (22-27 July, 2005).
37. M. Kipp, Anvil - A generic annotation tool for multimodal dialogue, in 7th Eur. Conf. Speech Communication and Technology (Eurospeech'2001), Aalborg, Denmark (3-7 September, 2001), pp. 1367-1370.
38. E. Douglas-Cowie, L. Devillers, J.-C. Martin, R. Cowie, S. Savvidou, S. Abrilian and C. Cox, Multimodal databases of everyday emotion: Facing up to complexity, in 9th Eur. Conf. Speech Communication and Technology (Interspeech'2005), Lisbon, Portugal (4-8 September, 2005), pp. 813-816.
39. L. Devillers, S. Abrilian and J.-C. Martin, Representing real life emotions in audiovisual data with non-basic emotional patterns and context features, in 1st Int. Conf. Affective Computing and Intelligent Interaction (ACII'2005), Beijing, China, 22-24 October (Springer-Verlag, Berlin, 2005), pp. 519-526.
40. S. Abrilian, L. Devillers and J.-C. Martin, Annotation of emotions in real-life video interviews: Variability between coders, in 5th Int. Conf. Language Resources and Evaluation (LREC'2006), Genoa, Italy (24-26 May, 2006).
41. D. McNeill, Hand and Mind - What Gestures Reveal about Thoughts (University of Chicago Press, IL, 1992).
42. A. Kendon, Gesture: Visible Action as Utterance (Cambridge University Press, 2004).
43. I. Poggi, Mind markers, in Gestures. Meaning and Use, eds. M. Rector, I. Poggi and N. Trigo (Edicoes Universidade Fernando Pessoa, Oporto, 2003), pp. 119-132.
44. S. Kaiser and T. Wehrle, Facial expressions as indicators of appraisal processes, in Appraisal Theories of Emotions: Theories, Methods, Research, eds. K. R. Scherer, A. Schorr and T. Johnstone (Oxford University Press, New York, 2001), pp. 285-300.
45. H. G. Wallbott and K. R. Scherer, Cues and channels in emotion recognition, J. Pers. Soc. Psychol. 51(4) (1986) 690-699.
46. P. Gallaher, Individual differences in nonverbal behavior: Dimensions of style, J. Pers. Soc. Psychol. 63 (1992) 133-145.
47. S. Kopp and I. Wachsmuth, Synthesizing multimodal utterances for conversational agents, J. Comput. Animat. Virtual Worlds 15(1) (2004) 39-52.
48. S. Kopp, P. Tepper and J. Cassell, Towards integrated microplanning of language and iconic gesture for multimodal output, in Int. Conf. Multimodal Interfaces (ICMI'04), Penn State University, State College, PA (14-15 October, 2004), pp. 97-104.
49. B. Hartmann, M. Mancini and C. Pelachaud, Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis, in Computer Animation (CA'2002), Geneva, Switzerland, 19-21 June (IEEE Computer Society, 2002), pp. 111-119.
50. B. Hartmann, M. Mancini and C. Pelachaud, Implementing expressive gesture synthesis for embodied conversational agents, in Gesture Workshop (GW'2005), Vannes, France, 18-20 May (Springer, 2005).
51. C. Pelachaud, Multimodal expressive embodied conversational agent, in ACM Multimedia, Brave New Topics Session (ACM, Singapore, 2005), pp. 683-689.
52. P. Ekman, The Face Revealed (Weidenfeld & Nicolson, London, 2003).
53. D. Keltner, Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame, J. Pers. Soc. Psychol. 68 (1995) 441-454.
54. D. Matsumoto, More evidence for the universality of a contempt expression, Motiv. Emotion 16, 363-368 (1992).
55. B. Bouchon-Meunier, M. Rifqi and S. Bothorel, Towards general measures of comparison of objects, Fuzzy Set. Sys. 84 (1996) 143-153.
56. D. Dubois and H. Prade, Fuzzy Sets and Systems (Academic Press, New York, 1980).
57. P. Ekman and W. V. Friesen, The repertoire of nonverbal behavior's: Categories, origins, usage and coding, Semiotica 1 (1969) 49-98.
58. P. Ekman, Darwin, deception, and facial expression, Ann. NY Acad. Sci. 1000 (2003) 205-221.
59. E. Hüllermeier, D. Dubois and H. Prade, Fuzzy rules in case-based reasoning, in Conf. AFIA-99 Raisonnement à Partir de Cas, Paris, France (17 June, 1999), pp. 45-54.
60. P. Ekman and W. V. Friesen, Manual for the Facial Action Coding System (Consulting Psychology Press, Palo Alto, CA, 1978).
61. S. Prillwitz, R. Leven, H. Zienert, T. Hanke and J. Henning, Hamburg notation system for sign languages: An introductory guide, in International Studies on Sign Language and Communication of the Deaf (Signum Press, Hamburg, Germany, 1989).
62. B. De Carolis, C. Pelachaud, I. Poggi and M. Steedman, APML, a markup language for believable behavior generation, in Life-Like Characters. Tools, Affective Functions and Applications, eds. H. Prendinger and M. Ishizuka (Springer, 2004), pp. 65-85.