Audio-visual spontaneous emotion recognition

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4451 LNAI, Issue , 2007, Pages 72-90

  Zeng, Zhihong ^a   Hu, Yuxiao ^a   Roisman, Glenn I ^a   Wen, Zhen ^b   Fu, Yun ^a   Huang, Thomas S ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Affect recognition; Affective computing; Emotion recognition; Multimodal human computer interaction

Indexed keywords

ARTIFICIAL INTELLIGENCE; GESTURE RECOGNITION; HIDDEN MARKOV MODELS; LEARNING SYSTEMS; MARKOV PROCESSES; OBJECT RECOGNITION; PROGRAMMING THEORY;

ADABOOST; ADABOOST LEARNING; ADULT ATTACHMENT INTERVIEW; AFFECT RECOGNITION; AFFECTIVE COMPUTING; AFFECTIVE STATES; AUDIO-VISUAL; CHALLENGING PROBLEM; EMOTION RECOGNITION; EMOTIONAL EXPRESSIONS; FACIAL ACTION CODING SYSTEM; FACIAL EXPRESSIONS; HUMAN COMPUTING; HYDERABAD , INDIA; INTERNATIONAL WORKSHOPS; JOINT CONFERENCE; MULTI-STREAM HIDDEN MARKOV MODEL; MULTIMODAL HUMAN-COMPUTER INTERACTION; MULTIMODAL RECOGNITION; VOCAL EXPRESSION;

FACE RECOGNITION;

EID: 49949108559     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-72348-6_4     Document Type: Conference Paper

References (83)

1. Picard, R.W., Affective Computing, MIT Press, Cambridge, 1997.
2. Litman, D.J. and Forbes-Riley, K., Predicting Student Emotions in Computer-Human Tutoring Dialogues. In Proc. of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), July 2004
3. Kapoor, A. and Picard, R.W., Multimodal Affect Recognition in Learning Environments, ACM Multimedia, 2005, 677-682
4. Viola P. 2004. Robust Real-Time Face Detection. Int. Journal of Computer Vision. 57(2), 137-154
5. Polzin, S.T. and Waibel, A. (1999), Pronunciation Variations in Emotional Speech, Proceedings of the ESCA Workshop, 103-108
6. Athanaselis, T., Bakamidis, S., Dologlou, I., Cowie, R., Douglas-Cowie, E., Cox, C. (2005), ASR for Emotional Speech: Clarifying the Issues and Enhancing Performance, Neural Networks, 18: 437-444
7. Steeneken, H.J.M. and Hansen, J.H.L. (1999), Speech under stress conditions: Overview of the effect of speech production and on system performance, Int. Conf. on Acoustics, Speech, and Signal Processing, 4:2079-2082
8. Okawa, S., Bocchieri, E. and Potamianos, A., Multi-band Speech Recognition in noisy environments, ICASSP, 1998, 641-644
9. Garg, A., Potamianos, G., Neti, C. & Huang, T.S., Frame-dependent multi-stream reliability indicators for audio-visual speech recognition, ICASSP, 2003.
10. Ekman P, Friesen WV, Ellsworth P. (1972). Emotion in the Human Face. Elmsford, NY: Pergamon.
11. Izard CE. 1971. The face of Emotion. New York: Appleton-Century_Crofts
12. Scherer (2004), Feelings integrate the central representation of appraisal-driven response organization in emotion. In Manstead, A.S.R., Frijda, N.H. & Fischer, A.H. (Eds.), Feelings and emotions, The Amsterdam symposium (pp. 136-157). Cambridge: Cambridge University Press, 136-157.
13. Ekman P, Friensen WV, Hager JC 2002. Facial Action Unit System. Published by A Human Face.
14. Ekman P and Rosenberg EL. 2005. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using Facial Action Coding System. 2nd Ed. Oxford University Express.
15. Russell JA, Bachorowski JA and Fernandez-Dols JM. 2003. Facial and Vocal Expressions of Emotion. Annual Review Psychology, 2003, 54:329-49
16. Ekman P. and Oster H. 1979. Facial Expressions of Emotion. Annual Review Psychology. 30:527-54
17. Roisman, G.I., Tsai, J.L., Chiang, K.S.(2004), The Emotional Integration of Childhood Experience: Physiological, Facial Expressive, and Self-reported Emotional Response During the Adult Attachment Interview, Developmental Psychology, Vol. 40, No. 5, 776-789
18. Cohn JF and Tronick EZ. 1988. Mother Infant Interaction: the sequence of dyadic states at three, six and nine months. Development Psychology, 23, 68-77
19. Fried E. 1976. The impact of nonverbal communication of facial affect on children's learning. PhD thesis, Rutgers University, New Brunswick, NJ
20. Ekman P, Matsumoto D, and Friesen WV. 2005. Facial Expression in Affective Disorders. In What the Face Reveals. Edited by Ekman P and Rosenberg EL. 429-439
21. Zeng, Z, Fu, Y., Roisman, G.I., Wen, Z., Hu, Y. and Huang, T.S., One-class classification on spontaneous facial expressions, Automatic Face and Gesture Recognition, 281 - 286, 2006
22. Bourlard, H. and Dupont, S., A new ASR approach based on independent processing and recombination of partial frequency bands, ICSLP 1996
23. Devillers, L., Vidrascu L. and Lamel L., Challenges in real-life emotion annotation and machine learning based detection, Neural Networks, 18(2005), 407-422
24. Ekman, P., Hager, J.C., Methvin, C.H. and Irwin, W., Ekman-Hager Facial Action Exemplars, unpublished, San Francisco: Human Interaction Laboratory, University of California
25. Kanade, T., Cohn, J., and Tian, Y. (2000), Comprehensive Database for Facial Expression Analysis, In Proceeding of International Conference on Face and Gesture Recognition, 46-53
26. Pantic, M., Valstar, M.F, Rademaker, R. and Maat, L. (2005), Web-based database for facial expression analysis, Int. Conf. on Multimedia and Expo
27. JAFFE: www.mic.atr.co.jp/-mlyons/jaffe.html
28. Chen, L.S, Joint Processing of Audio-Visual Informa-tion for the Recognition of Emotional Expressions in Human-Computer Interaction, PhD thesis, UIUC, 2000
29. Cowie, R., Douglas-Cowie E. and Cox, C., Beyond emotion archetypes: Databases for emotion modelling using neural networks, 18(2005), 371-388
30. Ekman, P. and Rosenberg, E. (Eds.), What the face reveals. NY: Oxford University, 1997
31. Cohn, J.F. and Schmidt, K.L.(2004), The timing of Facial Motion in Posed and Spontaneous Smiles, International Journal of Wavelets, Multiresolution and Information Processing, 2, 1-12
32. Valstar MF, Pantic M, Ambadar Z and Cohn JF. 2006. Spontaneous vs. Posed Facial Behavior: Automatic Analysis of Brow Actions. Int. Conf. on Multimedia Interfaces. 162-170
33. Ekman, P. (1994), Strong Evidence for Universais in Facial Expressions: A Reply to Russell's Mistaken Critique, Psychological Bulletin, 115(2): 268-287
34. Pantic M., Rothkrantz, L.J.M., Toward an affect-sensitive multimodal human-computer interaction, Proceedings of the IEEE, Vol. 91, No. 9, Sept. 2003, 1370-1390
35. Pantic, M., Sebe, N., Cohn, J.F. and Huang, T., Affective Multimodal Human-Computer Interaction, in Proc. ACM Int'l Conf. on Multimedia, November 2005, 669-676
36. Sebe, N., Cohen, I., Gevers, T., and Huang, T.S. (2005), Multimodal Approaches for Emotion Recognition: A Survey, In Proc. Of SPIE-IS&T Electronic Imaging, SPIE Vol 5670: 56-67
37. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., and Taylor, J.G., Emotion Recognition in Human-Computer Interaction, IEEE Signal Processing Magazine, January 2001, 32-80
38. Chen, L. and Huang, T. S., Emotional expressions in audiovisual human computer interaction, Int. Conf. on Multimedia & Expo 2000, 423-426
39. Chen, L., Huang, T. S., Miyasato, T., and Nakatsu, R., Multimodal human emotion/expression recognition, Int. Conf. on Automatic Face & Gesture Recognition 1998, 396-401
40. De Silva, L. C., and Ng, P. C., Bimodal emotion recognition, Int. Conf. on Automatic Face & Gesture Recognition 2000, 332-335
41. Yoshitomi, Y., Kim, S., Kawano, T., and Kitazoe, T., Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face, in Proc. ROMAN 2000, 178-183
42. Hoch, S., Althoff, F., McGlaun, G., Rigoll, G., Bimodal fusion of emotional data in an automotive environment, ICASSP, Vol. II, 1085-1088, 2005
43. Wang, Y. and Guan, L., Recognizing human emotion from audiovisual information, ICASSP, Vol. II, 1125-1128
44. Zeng, Z., Hu, Y., Liu, M., Fu, Y. and Huang, T.S., Training Combination Strategy of Multi-stream Fused Hidden Markov Model for Audio-visual Affect Recognition, in Proc. ACM Int'l Conf. on Multimedia, 2005, 65-68
45. Zeng, Z., Tu, J., Pianfetti , P., Liu, M., Zhang, T., et al., Audio-visual Affect Recognition through Multi-stream Fused HMM for HCI, Int. Conf. Computer Vision and Pattern Recognition. 2005: 967-972
46. Zeng, Z., Tu., J., Liu, M., Huang, T.S. (2005), Multi-stream Confidence Analysis for Audio-Visual Affect Recognition, the Int. Conf. on Affective Computing and Intelligent Interaction, 946-971
47. Zeng, Z., Zhang, Z., Pianfetti, B., Tu, J., and Huang, T.S. (2005), Audio-visual Affect Recognition in Activation-evaluation Space, Int. Conf. on Multimedia & Expo, 828-831.
48. Zeng, Z., Tu, J., Liu, M., Huang, T.S., Pianfetti, B., Levinson, S. (2007), Audio-visual Affect Recognition, IEEE Transactions on Multimedia, in press
49. Song, M., Bu, J., Chen, C., and Li, N., Audio-visual based emotion recognition - A new approach, Int. Conf. Computer Vision and Pattern Recognition. 2004, 1020-1025
50. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, CM. et al., Analysis of Emotion Recognition using Facial Expressions, Speech and Multimodal Information. 2004. Int. Conf. Multimodal Interfaces. 205-211
51. Hoch, S., Althoff, F., McGlaun, G., Rigoll, G., Bimodal fusion of emotional data in an automotive environment, ICASSP, Vol. II, 1085-1088, 2005
52. Wang, Y. and Guan, L., Recognizing human emotion from audiovisual information, ICASSP, Vol. II, 1125-1128
53. Go HJ, Kwak KC, Lee DJ, and Chun MG. 2003. Emotion recognition from facial image and speech signal. Int. Conf. of the Society of Instrument and Control Engineers. 28902895
54. Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., and Movellan, J.(2005), Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior, IEEE CVPR'05
55. Sebe, N., Lew, M.S., Cohen, I., Sun, Y., Gevers, T., Huang, T.S.(2004), Authentic Facial Expression Analysis, Int. Conf. on Automatic Face and Gesture Recognition
56. Zeng, Z., Fu, Y., Roisman, G.I., Wen, Z., Hu, Y., and Huang, T.S. (2006). Spontaneous Emotional Facial Expression Detection. Journal of Multimedia, 1(5): 1-8.
57. Valstar MF, Pantic M and Ambadar Z, and Cohn JF. 2006. Spontaneous vs. Posed Facial Behavior: Automatic Analysis of Brow Actions. Int. Conf. on Multimodal Interfaces. 162-170
58. Cohn JF, Reed LI, Ambadar Z, Xiao J, and Moriyama T. 2004. Automatic Analysis and recognition of brow actions and head motion in spontaneous facial behavior. Int. Conf. on Systems, Man & Cybernetics, 1, 610-616
59. Litman, D.J. and Forbes-Riley, K., Predicting Student Emotions in Computer-Human Tutoring Dialogues. In Proc. of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), July 2004
60. Batliner A, Fischer K, Hubera R, Spilkera J and Noth E. 2003. How to find trouble in communication. Speech Communication, Vol. 40, 117-143.
61. Neiberg D, Elenius K, and Laskowski K. 2006. Emotion Recognition in Spontaneous Speech Using GMM. Int. Conf. on Spoken Language Processing, 809-812
62. Ang J, Dhillon R, Krupski A, et al. 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog, ICSLP.
63. Fragopanagos, F. and Taylor, J.G., Emotion recognition in human-computer interaction, Neural Networks, 18 (2005) 389-405
64. Garidakis, G., Malatesta, L., Kessous, L., Amir, N., Paouzaiou, A. and Karpouzis, K. 2006. Modeling Naturalistic Affective States via Facial and Vocal Expression Recognition. Int. Conf. on Multimodal Interfaces. 146-154
65. Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., & Schröder, M. (2000). 'Feeltrace': an instrument for recording perceived emotion in real time. Proceedings of the ISCA Workshop on Speech and Emotion, 19-24
66. Maat L and Pantic M. 2006. Gaze-X: Adaptive Affective Multimodal Interface for Single-User Office Scenarios. Int. Conf. on Multimodal Interfaces. 171-178
67. Lanitis, A., Taylor, C. and Cootes, T. (1995), A Unified Approach to Coding and Interpreting Face Images, in Proc. International Conf. on Computer Vision, 368-373
68. Black, M. and Yacoob, Y.(1995), Tracking and Recognizing Rigid and Non-rigid Facial Motions Using Local Parametric Models of Image Motion, in Proc. Int. Conf. on Computer Vision, 374-381
69. Rosenblum, M., Yacoob, Y., and Davis, L. (1996), Human Expression Recognition from Motion Using a Radial Basis Function Network Architecture, IEEE Trans. On Neural Network, 7(5): 1121-1138
70. Essa, I. and Pentland, A. (1997), Coding, Analysis, Interpretation, and Recognition of Facial Expressions, IEEE Trans. On Pattern Analysis and Machine Intelligence, 19(7): 757-767
71. Cohen, L., Sebe, N., Garg, A., Chen, L., and Huang, T. (2003), Facial expression recognition from video sequences: Temporal and static modeling, Computer Vision and Image Understanding, 91(1-2): 160-187
72. Tian, Y., Kanade, T., Cohn, J.F. (2001), Recognizing Action Units for Facial Expression Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2): 97-115
73. Pantic M and Patras I. 2006. 'Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences', IEEE Transactions on Systems, Man and Cybernetics - Part B, vol. 36, no. 2, pp. 433-449
74. Kwon, O.W., Chan, K., Hao, J., Lee, T.W (2003), Emotion Recognition by Speech Signals, EUROSPEECH.
75. Polzin, Thomas (1999), Detecting Verbal and Non-verbal cues in the communication of emotion, PhD thesis, Carnegie Mellon University
76. Amir, N. and Ron, S. (1998), Toward Automatic Classification of Emotions in Speech, in Proc. ICSLP, 555-558
77. Dellaert, F., Polzin, T., and Waibel, A. (1996), Recognizing Emotion in Speech, In Proc. ICSLP, 1970-1973
78. Petrushin, V.A. (2000), Emotion Recognition in Speech Signal, In Proc. ICSLP, 222-225
79. Pantic M, Pentland A, Nijholt A and Huang TS. 2006. Human Computing and Machine Understanding of Human Behavior: A Survey. Int. Conf. Multimodal Interfaces. 233-238
80. Huang, D. (1999), Physiological, subjective, and behavioral Responses of Chinese American and European Americans during moments of peak emotional intensity, honor Bachelor thesis, Psychology, University of Minnesota.
81. Tao, H. and Huang, T.S., Explanation-based facial motion tracking using a piecewise Bezier volume deformation mode, IEEE CVPR'99, vol.1, pp. 611-617, 1999
82. Wen Z and Huang T. 2003. Capturing Subtle Facial Motions in 3D Face Tracking. Intl. Conf. on Computer Vision (ICCV). 1343-1350
83. He, X., Yan, S., Hu, Y., and Zhang, H, Learning a Locality Preserving Subspace for Visual Recognition, Int. Conf. on Computer Vision, 2003