Multimodal emotion recognition

Machine Audition: Principles, Algorithms and Systems

Volumn , Issue , 2010, Pages 398-423

  Haq, Sanaul ^a   Jackson, Philip J B ^a  

^a UNIVERSITY OF SURREY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84896883129     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4018/978-1-61520-919-4.ch017     Document Type: Chapter

References (101)

1. Anderson, K., & McOwan, P. W. (2006). A Real-Time Automated System for Recognition of Human Facial Expressions. IEEE Trans. Systems, Man, and Cybernetics Part B, 36(1), 96-105. doi:10.1109/TSMCB.2005.854502
2. Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K., et al. (2007). The Painful Face: Pain Expression Recognition Using Active Appearance Models. Proc. Ninth ACM Int'l Conf. Multimodal Interfaces (pp. 9-14).
3. Athanaselis, T., Bakamidis, S., Dologlou, I., Cowie, R., Douglas-Cowie, E., & Cox, C. (2005). ASR for Emotional Speech: Clarifying the Issues and Enhancing Performance. Neural Networks, 18, 437-444. doi:10.1016/j.neunet.2005.03.008
4. Bartlett, M. S., Littlewort, G., Braathen, P., Sejnowski, T. J., & Movellan, J. R. (2003). A Prototype for Automatic Recognition of Spontaneous Facial Actions. Advances in Neural Information Processing Systems, 15, 1271-1278.
5. Bartlett, M. S., Littlewort, G., Frank, M., et al. (2005). Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition (pp. 568-573).
6. Bartlett, M. S., Littlewort, G., Frank, M. G., et al. (2006). Fully Automatic Facial Action Recognition in Spontaneous Behavior. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition (pp. 223-230).
7. Batliner, A., et al. (2004). You Stupid Tin Box- Children Interacting with the AIBO Robot: A Cross-Linguistic Emotional Speech. Proc. Fourth Int'l Conf. Language Resources and Evaluation.
8. Batliner, A., Fischer, K., Hubera, R., Spilkera, J., & Noth, E. (2003). How to Find Trouble in Communication. Speech Communication, 40, 117-143. doi:10.1016/S0167-6393(02)00079-1
9. Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press.
10. Borchert, M., & Düsterhöft, A. (2005). Emotions in Speech - Experiments with Prosody and Quality Features in Speech for Use in Categorical and Dimensional Emotion Recognition Environments. Proc. IEEE Int'l Conf. on Natural Language Processing and Knowledge Engineering (pp. 147-151).
11. Burger, S., MacLaren, V., & Yu, H. (2002). The ISL Meeting Corpus: The Impact of Meeting Type on Speech Style. Proc. Eighth Int'l Conf. Spoken Language Processing.
12. Burges, C. J. C. (1998). A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery, 2, 121-167. doi:10.1023/A:1009715923555
13. Burkhardt, F., Paeschke, A., Rolfes, M., Send-lmeier, W., & Weiss, B. (2005). A Database of German Emotional Speech (pp. 1517-1520). Proc. Interspeech.
14. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., et al. (2004). Analysis of Emotion Recognition Using Facial Expressions. Speech and Multimodal Information. Proc. ACM Int'l Conf. Multimodal Interfaces (pp. 205-211).
15. Busso, C., & Narayanan, S. S. (2007). Interrelation between Speech and Facial Gestures in Emotional Utterances: A Single Subject Study. IEEE Trans. on Audio, Speech, and Language Processing, 15(8), 2331-2347. doi:10.1109/TASL.2007.905145
16. Campbell, J. (1997). Speaker Recognition: A Tutorial. Proceedings of the IEEE, 85(9), 1437-1462. doi:10.1109/5.628714
17. Caridakis, G., Malatesta, L., Kessous, L., Amir, N., Paouzaiou, A., & Karpouzis, K. (2006). Modeling Naturalistic Affective States via Facial and Vocal Expression Recognition. Proc. ACM Int'l Conf. Multimodal Interfaces, (pp. 146-154).
18. Chang, Y., Hu, C., Feris, R., & Turk, M. (2006). Manifold Based Analysis of Facial Expression. J. Image and Vision Computing, 24(6), 605-614. doi:10.1016/j.imavis.2005.08.006
19. Chang, Y., Vieira, M., Turk, M., & Velho, L. (2005). Automatic 3D Facial Expression Analysis in Videos. Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures (Vol. 3723, pp. 293-307).
20. Chen, C. (1978). Pattern Recognition and Signal Processing. The Netherlands: Sijthoff & Noordoff.
21. Chen, C., Huang, Y., & Cook, P. (2005). Visual/Acoustic emotion recognition. Proc. Int'l Conf. on Multimedia & Expo (pp. 1468-1471).
22. Chen, L. S. (2000). Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions in Human-Computer Interaction. (PhD dissertation, Univ. of Illinois, Urbana-Champaign).
23. Cohen, L., Sebe, N., Garg, A., Chen, L., & Huang, T. (2003). Facial Expression Recognition from Video Sequences: Temporal and Static Modeling. Computer Vision and Image Understanding, 91(12), 160-187. doi:10.1016/S1077-3142(03)00081X
24. Cohn, J. F. (2006). Foundations of Human Computing: Facial Expression and Emotion. Proc. Eighth ACM Int'l Conf. Multimodal Interfaces, (pp. 233-238).
25. Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J., & Moriyama, T. (2004). Automatic Analysis and Recognition of Brow Actions and Head Motion in Spontaneous Facial Behavior. Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics (vol. 1, pp. 610-616).
26. Cohn, J. F., & Schmidt, K. L. (2004). The Timing of Facial Motion in Posed and Spontaneous Smiles. International Journal of Wavelets, Multresolution, and Information Processing, 2, 1-12. doi:10.1142/S021969130400041X
27. Cowie, R., Douglas-Cowie, E., et al. (2000). Feeltrace: An Instrument for Recording Perceived Emotion in Real Time. Proc. ISCA Workshop Speech and Emotion (pp. 19-24).
28. Cowie, R., Douglas-Cowie, E., & Cox, C. (2005). Beyond Emotion Archetypes: Databases for Emotion Modeling Using Neural Networks. Neural Networks, 18, 371-388. doi:10.1016/j.neunet.2005.03.002
29. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine, 32-80. doi:10.1109/79.911197
30. Devillers, L. & Vidrascu, L. (2006). Real-Life Emotions Detection with Lexical and Paralinguistic Cues on Human-Human Call Center Dialogs. Proc. Interspeech, 801-804.
31. Douglas-Cowie, E., Campbell, N., Cowie, R., & Roach, P. (2003). Emotional Speech: Towards a New Generation of Database. Speech Communication, 40(1/2), 33-60. doi:10.1016/S01676393(02)00070-5
32. Duda, R., Hart, P., & Stork, D. (2001). Pattern Classification. New York: John Wiley & Sons, Inc.
33. Edwards, A. L. (1962). Experimental Design in Psychological Research. New York: Holt, Rinehart and Winston.
34. Ekman, P. (1971). Universal and Cultural Differences in Facial Expressions of Emotion. Proc. Nebraska Symp. Motivation, 207-283.
35. Ekman, P. (1994). Strong Evidence for Universals in Facial Expressions: A Reply to Russell's Mistaken Critique. Psychological Bulletin, 115(2), 268-287. doi:10.1037/0033-2909.115.2.268
36. Ekman, P., & Rosenberg, E. L. (2005). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System,(Eds.). Oxford Univ. Press.
37. Engberg, I. S., & Hansen, A. V. (1996). Documentation of the Danish Emotional Speech Database (DES). Denmark: Aalborg University.
38. Forbes-Riley, K., & Litman, D. (2004). Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources. Proc. Human Language Technology Conf. North Am. Chapter of the Assoc. Computational Linguistics (HLT/NAACL).
39. Fragopanagos, F., & Taylor, J. G. (2005). Emotion Recognition in Human-Computer Interaction. Neural Networks, 18, 389-405. doi:10.1016/j.neunet.2005.03.006
40. Freund, Y., & Schapire, R. E. (1999). A Short Introduction to Boosting. Journal of Japanese Society for Artificial Intelligence, 14(5), 771-780.
41. Graciarena, M., Shriberg, E., Stolcke, A., Enos, F., Hirschberg, J., & Kajarekar, S. (2006). Combining Prosodic, Lexical and Cepstral Systems for Deceptive Speech Detection. Proc. Int'l Conf. Acoustics, Speech and Signal Processing, 1, 1033-1036.
42. Gunes, H., & Piccardi, M. (2005). Affect Recognition from Face and Body: Early Fusion versus Late Fusion. Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics (pp. 3437-3443).
43. Gunes, H., & Piccardi, M. (2006). A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior. Proc. 18th Int'l Conf. Pattern Recognition, 1, 1148-1153.
44. Guo, G., & Dyer, C. R. (2005). Learning from Examples in the Small Sample Case: Face Expression Recognition. IEEE Trans. Systems, Man, and Cybernetics Part B, 35(3), 477-488. doi:10.1109/TSMCB.2005.846658
45. Hager, J. C. (2003). Date Face. Retrieved from http://face-and-emotion.com/dataface/general/homepage.jsp
46. Haq, S., & Jackson, P. J. B. (2009). Speaker-Dependent Audio-Visual Emotion Recognition. Proc. Auditory-Visual Speech Processing.
47. Haq, S., Jackson, P. J. B., & Edge, J. (2008). Audio-visual feature selection and reduction for emotion Classification (pp. 185-190). Proc. Auditory-Visual Speech Processing.
48. Kanade, T., Cohn, J., & Tian, Y. (2000). Comprehensive Database for Facial Expression Analysis, Proc. IEEE Int'l Conf. Face and Gesture Recognition (pp. 46-53).
49. Kao, Y., & Lee, L. (2006). Feature Analysis for Emotion Recognition from Mandarin Speech Considering the Special Characteristics of Chinese Language (pp. 1814-1817). Proc. Interspeech.
50. Kotsia, I., & Pitas, I. (2007). Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines. IEEE Transactions on Image Processing, 16(1), 172-187. doi:10.1109/TIP.2006.884954
51. Lee, C. M., & Narayanan, S. S. (2005). Toward Detecting Emotions in Spoken Dialogs. IEEE Transactions on Speech and Audio Processing, 13(2), 293-303. doi:10.1109/TSA.2004.838534
52. Lin, Y., & Wei, G. (2005). Speech Emotion Recognition Based on HMM and SVM. Proc. 4th Int'l Conf. on Mach. Learn. and Cybernetics (pp.4898-4901).
53. Litman, D. J., & Forbes-Riley, K. (2004). Predicting Student Emotions in Computer-Human Tutoring Dialogues. Proc. 42nd Ann. Meeting of the Assoc. Computational Linguistics.
54. Littlewort, G. C., Bartlett, M. S., & Lee, K. (2007). Faces of Pain: Automated Measurement of Spontaneous Facial Expressions of Genuine and Posed Pain, Proc. Ninth ACM Int'l Conf. Multimodal Interfaces, pp. 15-21.
55. Lucey, S., Ashraf, A. B., & Cohn, J. F. (2007). Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face. In Delac, K., & Grgic, M. (Eds.), Face Recognition (pp. 275-286). New York: I-Tech Education and Publishing.
56. Luengo, I., & Navas, E. (2005). Automatic Emotion Recognition using Prosodic Parameters (pp. 493-496). Proc. Interspeech.
57. Neilberg, D., & Elenius, K. (2006). Emotion Recognition in Spontaneous Speech Using GMMs (pp. 809-812). Proc. Interspeech.
58. Nogueiras, A., Moreno, A., Bonafonte, A., & Mariño, J. B. (2001). Speech Emotion Recognition Using Hidden Markov Models (pp. 2679-2682). Proc. Eurospeech.
59. O'Toole, A. J. (2005). A Video Database of Moving Faces and People. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 812-816. doi:10.1109/TPAMI.2005.90
60. Ortony, A., & Turner, T. J. (1990). What's Basic About Basic Emotions? Psychological Review, 97(3), 315-331. doi:10.1037/0033-295X.97.3.315
61. Pal, P., Iyer, A. N., & Yantorno, R. E. (2006). Emotion Detection from Infant Facial Expressions and Cries, Proc. IEEE Int'l Conf. Acoustics, Speech and Signal Processing, 2, 721-724.
62. Pantic, M., & Bartlett, M. S. (2007). Machine Analysis of Facial Expressions. In Delac, K., & Grgic, M. (Eds.), Face Recognition (pp. 377-416). New York: I-Tech Education and Publishing.
63. Pantic, M., & Patras, I. (2006). Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments Form Face Profile Image Sequences. IEEE Trans. Systems, Man, and Cybernetics Part B, 36(2), 433-449. doi:10.1109/TSMCB.2005.859075
64. Pantic, M., & Rothkrantz, L. J. M. (2004). Facial Action Recognition for Facial Expression Analysis from Static Face Images. IEEE Trans. Systems, Man, and Cybernetics Part B, 34(3), 1449-1461. doi:10.1109/TSMCB.2004.825931
65. Pantic, M., Valstar, M. F., Rademaker, R., & Maat, L. (2005). Web-Based Database for Facial Expression Analysis. Proc. 13th ACM Int'l Conf. Multimedia, (pp. 317-321).
66. Petridis, S., & Pantic, M. (2008). Audiovisual Discrimination between Laughter and Speech. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (pp. 5117-5120).
67. Roisman, G. I., Tsai, J. L., & Chiang, K. S. (2004). The Emotional Integration of Childhood Experience: Physiological, Facial Expressive, and Self-Reported Emotional Response during the Adult Attachment Interview. Developmental Psychology, 40(5), 776-789. doi:10.1037/00121649.40.5.776
68. Russell, J., Ward, L., & Pratt, G. (1981). Affective Quality Attributed to Environments: A Factor Analytic Study. Environment and Behavior, 13(3), 259-288. doi:10.1177/0013916581133001
69. Scherer, K. (2005). What are emotions? and how can they be measured? Social Sciences Information. Information Sur les Sciences Sociales, 44(4), 695-729. doi:10.1177/0539018405058216
70. Schuller, B., Muller, R., Hornler, B., Hothker, A., Konosu, H., & Rigoll, G. (2007). Audiovisual Recognition of Spontaneous Interest within Conversations Proc. ACM Int'l Conf. Multimodal Interfaces (pp. 30-37).
71. Schuller, B., Rigoll, G., & Lang, M. (2003). Hidden Markov Model-Based Speech Emotion Recognition. Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, 2, 1-4.
72. Schuller, B., Villar, R. J., Rigoll, G., & Lang, M. (2005). Meta-Classifiers in Acoustic and Linguistic Feature Fusion-Based Affect Recognition. Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, (pp. 325-328).
73. Schuller, B., & Wimmer, M. (2008). Detection of security related affect and behavior in passenger Transport (pp. 265-268). Proc. Interspeech.
74. Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006). Emotion Recognition Based on Joint Visual and Audio Cues. Proc. Int'l Conf. Pattern Recognition (pp. 1136-1139).
75. Sebe, N., Lew, M. S., Cohen, I., Sun, Y., Gevers, T., & Huang, T. S. (2004). Authentic Facial Expression Analysis. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition.
76. Sethu, V., Ambikairajah, E., & Epps, J. (2008). Phonetic and speaker variations in automatic emotion Classification (pp. 617-620). Proc. Interspeech.
77. Shlens, J. (2005). A Tutorial on Principal Component Analysis. Systems Neurobiology Laboratory, Salk Institute for Biological Studies, La Jolla.
78. Song, M., Bu, J., Chen, C., & Li, N. (2004). Audio-Visual-Based Emotion Recognition: A New Approach. Proc. Int'l Conf. Computer Vision and Pattern Recognition, (pp. 1020-1025).
79. Swerts, M., & Krahmer, E. (2008). Gender-related differences in the production and perception of Emotion (pp. 334-337). Proc. Interspeech.
80. Tao, H., & Huang, T. S. (1999). Explanation-Based Facial Motion Tracking Using a Piecewise Bezier Volume Deformation Mode. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 1, 611-617.
81. Tian, Y. L., Kanade, T., & Cohn, J. F. (2005). Facial Expression Analysis. S.Z. Li and A.K. Jain (Eds.). Handbook of Face Recognition (pp. 247-276). New York: Springer.
82. Valstar, M., Pantic, M., Ambadar, Z., & Cohn, J. F. (2006). Spontaneous versus Posed Facial Behavior: Automatic Analysis of Brow Actions. Proc. Int'l Conf. Multimodal Interfaces (pp.162-170).
83. Valstar, M., Pantic, M., & Patras, I. (2004). Motion History for Facial Action Detection from Face Video. Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, 1, 635-640.
84. Valstar, M. F., Gunes, H., & Pantic, M. (2007). How to Distinguish Posed from Spontaneous Smiles Using Geometric Features. Proc. ACM Int'l Conf. Multimodal Interfaces (pp. 38-45).
85. Ververidis, D., & Kotropoulos, C. (2005). Emotional speech classification using Gaussian mixture Models. Proc. ISCAS (pp. 2871-2874).
86. Vidrascu, L., & Devillers, L. (2005). Detection of real-life emotions in call centers (pp. 1841-1844). Proc. Interspeech.
87. Wang, J., Yin, L., Wei, X., & Sun, Y. (2006). 3D Facial Expression Recognition Based on Primitive Surface Feature Distribution. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2, 1399-1406.
88. Wang, Y., & Guan, L. (2005). Recognizing Human Emotion from Audiovisual Information. Proc. Int'l Conf. Acoustics, Speech, and Signal Processing (pp. 1125-1128).
89. Wen, Z., & Huang, T. S. (2003). Capturing Subtle Facial Motions in 3D Face Tracking. Proc. Ninth IEEE Int'l Conf. Computer Vision (pp. 1343-1350).
90. Whissell, C. M. (1989). The Dictionary of Affect in Language, Emotion: Theory, Research and Experience. In Plutchik, R., & Kellerman, H. (Eds.), The Measurement of Emotions (Vol. 4, pp. 113-13). Academic Press.
91. Whitehill, J., & Omlin, C. W. (2006). Haar Features for FACS AU Recognition, Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition (pp. 217-222).
92. Witten, I. H., & Frank, E. (2000). Data Mining: Practical machine learning tools with java implementations. San Francisco: Morgan Kaufmann.
93. Xiao, J., Moriyama, T., Kanade, T., & Cohn, J. F. (2003). Robust Full-Motion Recovery of Head by Dynamic Templates and Re-Registration Techniques. International Journal of Imaging Systems and Technology, 13(1), 85-94. doi:10.1002/ima.10048
94. Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006). A 3D Facial Expression Database for Facial Behavior Research. Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition(pp.211-216).
95. Young, S., & Woodland, P. (2009). Hidden Markov Model Toolkit. Cambridge University Engineering Department (CUED), UK. Online: http://htk.eng.cam.ac.uk/.
96. Zeng, Z., Fu, Y., Roisman, G. I., Wen, Z., Hu, Y., & Huang, T. S. (2006). Spontaneous Emotional Facial Expression Detection. J. Multimedia, 1(5), 1-8.
97. Zeng, Z., Hu, Y., Roisman, G. I., Wen, Z., Fu, Y., & Huang, T. S. (2007a). Audio-Visual Spontaneous Emotion Recognition. In Huang, T. S., Nijholt, A., Pantic, M., & Pentland, A. (Eds.), Artificial Intelligence for Human Computing (pp. 72-90). New York: Springer. doi:10.1007/978-3-54072348-6_4
98. Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S. (2007b). Audio-Visual Affect Recognition. IEEE Transactions on Multimedia, 9(2), 424-428. doi:10.1109/TMM.2006.886310
99. Zeng, Z., Tu, J., Pianfetti, P., Liu, M., Zhang, T., Zhang, Z., et al. (2005a). Audio-Visual Affect Recognition through Multi-Stream Fused HMM for HCI, Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition (pp. 967-972).
100. Zeng, Z., Zhang, Z., Pianfetti, B., Tu, J., & Huang, T. S. (2005b). Audio-Visual Affect Recognition in Activation-Evaluation Space. Proc. 13th ACM Int'l Conf. Multimedia (pp. 828-831).
101. Zhang, Y., & Ji, Q. (2005). Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(5), 699-714. doi:10.1109/TPAMI.2005.93