Recognizing human emotional state from audiovisual signals

IEEE Transactions on Multimedia

Volumn 10, Issue 4, 2008, Pages 659-668

  Wang, Yongjin ^a   Guan, Ling ^b  

^a UNIVERSITY OF TORONTO   (Canada)

^b RYERSON UNIVERSITY   (Canada)

Author keywords

Audiovisual information; Emotion recognition; Multiclassifier

Indexed keywords

ALGORITHMS; CLASSIFICATION (OF INFORMATION); FEATURE EXTRACTION; SIGNAL ANALYSIS; SPEECH RECOGNITION; WAVELET ANALYSIS;

AUDIOVISUAL INFORMATION; EMOTION RECOGNITION; MULTICLASSIFIER;

GESTURE RECOGNITION;

EID: 44049094896     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2008.921734     Document Type: Article

References (36)

1. R. W. Picard, Affective Computing. Cambridge, MA: MIT Press, 1997.
2. P. Ekman and M. O'Sullivan, "The role of context in interpreting facial expression: Comment on Russell and Fehr," J. Exper. Psychol., General, vol. 117, pp. 86-88, 1987.
3. P. Ekman, "Strong evidence for universais in facial expressions: A reply to Russell's mistaken critique," Psychol. Bull., vol. 115, pp. 268-287, 1994.
4. L. C. De Silva, T. Miyasato, and R. Nakatsu, "Facial emotion recognition using multi-modal information," in Proc. IEEE Int. Conf. Information, Communications and Signal Processing, Singapore, Sep. 1997, vol. 1, pp. 397-401.
5. M. Song, C. Chen, and M. You, "Audio-visual based emotion recognition using triples hidden Markov model," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Montreal, QC, Canada, May 2004, vol. 5, pp. 877-880.
6. M. Song, J. Bu, C. Chen, and N. Li, "Audio-visual based emotion recognition: A new approach," in Proc. IEEE Comput. Soc Conf. Computer Vision and Pattern Recognition, 2004, vol. 2, pp. 1020-1025.
7. L. C. De Silva and P. C. Ng, "Bimodal emotion recognition," in Proc. 4th IEEE Int. Conf. on Automatic Face and Gesture Recognition, France, Mar. 2000, pp. 332-335.
8. L. S. Chen, H. Tao, T. S. Huang, T. Miyasato, and R. Nakatsu, "Emotion recognition from audiovisual information," in Proc. IEEE 2nd Workshop on Multimedia Signal Processing, CITY?, CA, Dec. 1998, pp. 83-88.
9. C. Chen, Y. Huang, and P. Cook, "Visual/acoustic emotion recognition," in Proc. IEEE Int. Conf. Multimedia and Expo, 2005, pp. 1468-1471.
10. S. Hoch, F. Althoff, G. McGlaun, and G. Rigoll, "Bimodal fusion of emotional data in an automotive environment," in Proc. IEEE Int. Conf. on Acoustic, Speech, and Signal Processing, 2005, vol. 2, pp. 1085-1088.
11. Z. Zeng et al., "Bimodal HCI-related affect recognition," in Proc. 6th Int. Conf. Multimodal Interfaces, (ICMI'04), 2004, pp. 137-143.
12. Z. Zeng et al, "Audio-visual affect recognition through multi-stream fused HMM for HCI," in IEEE Comput. Soc. Conf. Computer Vision and Pattern Recognition (CVPR '05), 2005, vol. 2, pp. 967-972.
13. G. Bailly, C. Benoit, and T. R. Sawallis, Talking Machines: Theories, Models, and Designs. Amsterdam, The Netherlands: Elsevier, 1992.
14. F. Dellaert, T. Polzin, and A. Waibel, "Recognizing emotion in speech," in Proc. 4th Int. Conf. on Spoken Language Processing, Philadelphia, PA, Oct. 1996, vol. 3, pp. 1970-1973.
15. C. M. Lee, S. S. Narayanan, and R. Pieraccini, "Classifying emotions in human-machine spoken dialogs," in Proc. Int. Conf. on Multimedia and Expo, Switzerland, Aug. 2002, vol. 1, pp. 737-740.
16. D. Ververidis, C. Kotropoulos, and I. Pitas, "Automatic emotional speech classification," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Montreal, QC, Canada, May 2004, vol. 1, pp. 593-596.
17. T. L. Nwe, F. S. Wei, and L. C. De Silva, "Speech based emotion classification," in Proc. IEEE Region 10 Conf. Electrical and Electronics Technology, Singapore, Aug. 2001, vol. 1, pp. 297-301.
18. O. Kwon, K. Chan, J. Hao, and T. Lee, "Emotion recognition by speech signals," in Proc. Euro. Conf. on Speech Comm. and Tech., Geneva, Switzerland, Sept. 2003, pp. 125-128.
19. T. L. Nwe, F. S. Wei, and L. C. De Silva, "Stress classification using subband features," IEICE Trans. Inform. Syst., vol. E86-D, no. 3, pp. 105-116,2003.
20. A. Bartlett, V. Evans, I. Frenkel, C. Hobson, and E. Sumera, Digital Hearing Aids [Online]. Available: www.owlnet.rice.edu/~ elec301/Projects01/ dig_hear_ aid/2004
21. Introduction to Computer Programming With MATLAB Department of Phonetics and Linguistics, Univ. College Landon [Online]. Available: www.phon.ucl.ac.uk/ courses/spsci/matlab/, 2004
22. C. Becchetti and L. P. Ricotti, Speech Recognition: Theory and C++ Implementation. Toronto, ON, Canada: Wiley, 1999.
23. M. J. Lyons, J. Budynek, A. Plante, and S. Akamatsu, "Classifying facial attributes using a 2-D Gabor wavelet representation and discriminant analysis," in Proc. 4th Int. Conf. Automatic Face and Gesture Recognition, France, Mar. 2000, pp. 202-207.
24. L. Ma and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," IEEE Trans. Syst., Man, Cybern, B:Cybern., vol. 34, pp. 1588-1595, 2004.
25. D. Kim and Z. Bien, "Fuzzy neural networks(FNN)-based approach for personalized facial expression recognition with novel feature selection method," in Proc. IEEE Int. Conf. on Fuzzy Systems, St. Louis, MO, May 2003, vol. 2, pp. 908-913.
26. M. Pantic and L. J. M. rothkrantz, "Facial action recognition for facial expression analysis from static face image," IEEE Trans. Syst., Man, Cybern B:Cybern., vol. 34, pp. 1449-1461, 2004.
27. L. C. De Silva and S. C. Hui, "Real-time facial feature extraction and emotion recognition," in Proc. 4th Int. Conf. on Information, Communications and Signal Processing, Singapore, Dec. 2003, vol. 3, pp. 1310-1314.
28. I. Cohen, N. Sebe, Y. Sun, M. S. Lew, and T. S. Huang, "Evaluation of expression recognition techniques," in Proc. of Int. Conf. on Image and Video Retrieval, Urbana, IL, Jul. 2003, pp. 184-195.
29. C Garcia and G. Tziritas, "Face detection using quantized skin color regions merging and wavelet packet analysis," IEEE Trans. Multimedia, vol. 1, no. 3, pp. 264-277, Sept. 1999.
30. B. S. Manjunath and W. Y. Ma, "Texture features for browsing and retrieval of image data," IEEE Trans. Pattern Anal. Machine Intell, vol. 18, pp. 837-842, Aug. 1996.
31. S. Theodorids and K. Koutroumbas, Pattern Recognition, 2nd ed. New York: Elsevier, 2003.
32. S. Haykin, Neural Networks: A Comprehensive Foundation. Upper Saddale River, NJ: Prentice-Hall, 1999.
33. J. Ghosh, "Multiclassifier systems: Back to the future, in multiple classifier systems," in Lecture Notes in Computer Science 2002, vol. 2364, pp. 1-15.
34. L. K. Hansen and P. Salamon, "Neural network ensembles," IEEE Trans. Pattern Anal. Machine Intell, vol. 12, pp. 993-1000, 1990.
35. J. Kittler, M. Hatef, R. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Machine Intell., vol. 20, pp. 226-239, 1998.
36. T. G. Dietterich, "Ensembel methods in machine learning," in Multiple Classifier Systems, J. Kittler and T. Roli, Eds. Berlin, Germany: Springer, 2001, vol. 1857, LNCS, pp. 1-15.