Audio-visual affective expression recognition through multistream fused HMM

IEEE Transactions on Multimedia

Volumn 10, Issue 4, 2008, Pages 570-577

  Zeng, Zhihong ^a,b   Tu, Jilin ^b   Pianfetti Jr , Brian M ^b   Huang, Thomas S ^a,b  

^a IEEE

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Affect recognition; Affective computing; Emotion recognition; Human computing; Human computer interaction; Multimodal fusion

Indexed keywords

ALGORITHMS; HUMAN COMPUTER INTERACTION; INFORMATION ANALYSIS; MARKOV PROCESSES;

AFFECT RECOGNITION; AFFECTIVE COMPUTING; EMOTION RECOGNITION; MULTIMODAL FUSION;

GESTURE RECOGNITION;

EID: 44049099067     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2008.921737     Document Type: Article

References (45)

1. N. Ambady and R. Rosenthal, "Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis," Psychol. Bull., vol. 111, no. 2, pp. 256-274, 1992.
2. M. Brand and N. Oliver, "Coupled hidden Markov models for complex action recognition," in Proc. Computer Vision Pattern Recognition, 1997, pp. 201-206.
3. C. Busso, Z. Deng, S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, "Analysis of emotion recognition using facial expressions speech and multimodal information," in Int. Conf. Multimodal Interfaces, 2004, pp. 205-211.
4. L. S. Chen, "Joint Processing of Audio-Visual Information for the Recognition of Emotional Expressions in Human-Computer Interaction," PhD thesis, UIUC, 2000.
5. L. Chen and T. S. Huang, "Emotional expressions in audiovisual human computer interaction," in Int. Conf. on Multimedia & Expo 2000, 2000, pp. 423-4-26.
6. L. Chen, T. S. Huang, T. Miyasato, and R. Nakatsu, "Multimodal human emotion/expression recognition," in Int. Conf. on Automatic Face & Gesture Recognition, 1998, pp. 396-401.
7. J. F. Cohn, "Foundations of human computing: Facial expression and emotion," in Int. Conf. on Multimodal Interfaces, 2006, pp. 233-238.
8. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., pp. 32-80, Jan. 2001.
9. L. C. De Silva and P. C. Ng, "Bimodal emotion recognition," in Int. Conf. on Automatic Face & Gesture Recognition 2000, 2000, pp. 332-335.
10. P. Ekman, "Universals and cultural differences in facial expressions of emotion," in Nebr. Symp. Motiv. 1971, 1972, pp. 207-283.
11. P. Ekman, D. Matsumoto, and W. V. Friesen, "Facial expression in affective disorders," in What the Face Reveals, P. Ekman and E. L. Rosenberg, Eds.:, 2005, pp. 429-439.
12. F. Fragopanagos and J. G. Taylor, "Emotion recognition in human-computer interaction," Neural Netw., vol. 18, pp. 389-405, 2005.
13. G. Caridakis, L. Malatesta, L. Kessous, N. Amir, A. Paouzaiou, and K. Karpouzis, "Modeling naturalistic affective states via facial and vocal expression recognition," in Int. Conf. on Multimodal Interfaces, 2006, pp. 146-154.
14. H. J. Go, K. C. Kwak, D. J. Lee, and M. G. Chun, "Emotion recognition from facial image and speech signal," in Int. Conf. Soc. Instrument and Control Engineers, 2003, pp. 2890-2895.
15. S. Hoch, F. Althoff, G. McGlaun, and G. Rigoll, "Bimodal fusion of emotional data in an automotive environment," in ICASSP, 2005, vol. II, pp. 1085-1088.
16. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. Int. Conf. Face and Gesture Recognition, 2000, pp. 46-53.
17. O. W. K won, K. Chan, J. Hao, and T. W. Lee, "Emotion recognition by speech signals," in EUROSPEECH 2003, 2003.
18. S. P. Luttrell, "The use of Bayesian and entropic methods in neural network theory," in Maximum Entropy and Bayesian Methods. Boston, MA: Kluwer, 1989, pp. 363-370.
19. A. Mehrabian, "Communication without words," Psychol. Today, vol. 2, no. 4, pp. 53-56, 1968.
20. H. Pan, Z. P. Liang, and T. S. Huang, "Estimation of the joint probability of multisensory signals," Pattern Recognit. Lett., vol. 22, pp. 1432-1437, 2001.
21. H. Pan, S. Levinson, T. S. Huang, and Z. P. Liang, "A fused hidden Markov model with application to bimodal speech processing," IEEE Trans. Signal Process., vol. 52, no. 3, pp. 573-581, March 2004.
22. M. Pantic and L. J. M. Rothkrantz, "Automatic analysis of facial expressions-the state of the art," IEEE Trans. Pattern Anal Mchine Intell., vol. 22, no. 12, pp. 1424-1445, 2000.
23. M. Pantic and L. J. M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sep. 2003.
24. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Machine Intell., to be published.
25. M. Pantic, N. Sebe, J. F. Cohn, and T. Huang, "Affective multimodal human-computer interaction," in Proc. ACM Int. Conf. on Multimedia, 2005, pp. 669-676.
26. M. Pantic, A. Pentland, A. Nijholt, and T. S. Huang, "Human computing and machine understanding of human behavior: A survey," in Int. Conf. on Multimodal Interfaces, 2006, pp. 239-248.
27. M. Pantic and M. S. Bartlett, "Machine analysis of facial expressions," in Face Recognition, K. Kurihara, Ed. Vienna, Austria: Advanced Robotics System, 2007, pp. 327-366.
28. R. W. Picard, Affective Computing. Cambridge: MIT Press, 1997.
29. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, no. 2, Feb. 1989.
30. A. Samal and P. A. Iyengar, "Automatic recognition and analysis of human faces and facial expressions: A survey," Pattern Recognit., vol. 25, no. 1, pp. 65-77, 1992.
31. L. K. Saul and M. I. Jordan, "Mixed memory Markov model: Decomposing complex stochastic processes as mixture of simpler ones," Mach. Learn., vol. 37, pp. 75-88, Oct. 1999.
32. B. Stein and M. A. Meredith, The Merging of Senses. Cambridge, MA: MIT Press, 1993.
33. M. Song, J. Bu, C Chen, and N. Li, "Audio-visual based emotion recognition-A new approach," in Int. Conf. Computer Vision and Pattern Recognition, 2004, pp. 1020-1025.
34. D. Talkin, "A robust algorithm for pitch tracking," in Speech Coding Synthesis, W. B. Kkeijn and K. K. Paliwal, Eds. Amsterdam, The Netherlands: Elsevier, 1995.
35. H. Tao and T. S. Huang, "Explanation-based facial motion tracking using a piecewise Bezier volume deformation mode," in CVPR'99, 1999, vol. 1, pp. 611-617.
36. Y. L. Tian, T. Kanade, and J. F. Cohn, "Facial expression analysis," in Handbook of Face Recognition, S. Z. Li and A. K. Jain, Eds. New York: Springer, 2005, pp. 247-276.
37. J. Tu, Z. Zhang, Z. Zeng, and T. S. Huang, "Face localization via hierarchical condensation with fisher boosting feature selection," in Proc. Computer Vision and Pattern Recognition, 2004, pp. 719-724.
38. Y. Wang and L. Guan, "Recognizing human emotion from audiovisual information," in ICASSP, 2005, vol. II, pp. 1125-1128.
39. Y. Yoshitomi, S. Kim, T. Kawano, and T. Kitazoe, "Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face," in Proc. ROMAN 2000, 2000, pp. 178-183.
40. Z. Zeng, J. Tu, M. Liu, T. Zhang, N. Zhang, Z. Zhang, T. S. Huang, D. Roth, and S. Levinson, "Bimodal HCI-related affect recognition," in Int. Conf. on Multimodal Interfaces, 2004, pp. 137-143.
41. Z. Zeng, J. Tu, M. Liu, and T. S. Huang, "Multi-stream confidence analysis for audio-visual affect recognition," in Int. Conf. on Affective Computing and Intelligent Interaction, 2005, pp. 946-971.
42. Z. Zeng, Z. Zhang, B. Pianfetti, J. Tu, and T. S. Huang, "Audio-visual affect recognition in activation-evaluation space," in Int. Conf. on Multimedia & Expo, 2005, pp. 828-831.
43. Z. Zeng, Y. Hu, G. I. Roisman, Z. Wen, Y. Fu, and T. S. Huang, "Audiovisual emotion recognition in adult attachment interview," in Int. Conf. Multimodal Interfaces, 2006, pp. 139-145.
44. Z. Zeng, J. Tu, M. Liu, T. S. Huang, B. Pianfetti, D. Roth, and S. Levinson, " Audio-visual affect recognition," IEEE Trans. Multimedia, vol. 9, no. 2, pp. 424-428, Feb. 2007.
45. Z. Zeng, J. Tu, B. Pianfetti, M. Liu, T. Zhang, Z. Zhang, T. S. Huang, and S. Levinson, "Audio-visual emotion recognition through multistream fused HMM for HCI applications," in Conf. on Computer Vision and Pattern Recognition (CVPR), 2005, no. 2, pp. 967-972.