Multimodal approaches for emotion recognition: A survey

Proceedings of SPIE - The International Society for Optical Engineering

Volumn 5670, Issue , 2005, Pages 56-67

  Sebe, Nicu ^a   Cohen, Ira ^b   Gevers, Theo ^a   Huang, Thomas S ^c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b HEWLETT PACKARD LABORATORIES   (United States)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

Emotion recognition; Human computer interaction; Multimodal approach

Indexed keywords

COMPUTER GRAPHICS; COMPUTER KEYBOARDS; COMPUTER VISION; DATA PROCESSING; MATHEMATICAL MODELS; MICE (COMPUTER PERIPHERALS); PROBABILITY; SENSITIVITY ANALYSIS;

EMOTION RECOGNITION; HUMAN COMPUTER INTELLIGENT INTERACTION (HCII); HUMAN-HUMAN INTERACTION; MULTIMODAL APPROACH;

HUMAN COMPUTER INTERACTION;

EID: 21944433750     PISSN: 0277786X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1117/12.600746     Document Type: Conference Paper

References (47)

1. B. Reeves and C. Nass, The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places, Cambridge Univ. Press, 1996.
2. R. W. Picard, Affective Computing, MIT Press, Cambridge, MA, 1997.
3. P. Ekman, "Strong evidence for Universals in facial expressions: A reply to Russell's mistaken critique," Psychological Bulletin 115(2), pp. 268-287, 1994.
4. P. Ekman and W. Friesen, Facial Action Coding System: Investigator's Guide, Consulting Psychologists Press, 1978.
5. K. Mase, "Recognition of facial expression from optical flow," IEICE Trans. E74(10), pp. 3474-3483, 1991.
6. N. Ueki, S. Morishima, H. Yamada, and H. Harashima, "Expression analysis/synthesis system based on emotion space constructed by multilayered neural network," Systems and Computers in Japan 25(13), pp. 95-103, 1994.
7. A. Lanitis, C. Taylor, and T. Cootes, "A unified approach to coding and interpreting face images," in Proc. International Conf. on Computer Vision, pp. 368-373, 1995.
8. M. Black and Y. Yacoob, "Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion," in Proc. International Conf. on Computer Vision, pp. 374-381, 1995.
9. M. Rosenblum, Y. Yacoob, and L. Davis, "Human expression recognition from motion using a radial basis function network architecture," IEEE Trans. on Neural Network 7(5), pp. 1121-1138, 1996.
10. I. Essa and A. Pentland, "Coding, analysis, interpretation, and recognition of facial expressions," IEEE Trans. on Pattern Analysis and Machine Intelligence 19(7), pp. 757-763, 1997.
11. T. Otsuka and J. Ohya, "Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences," in Proc. International Conf. on Image Processing, pp. 546-549, 1997.
12. I. Cohen, N. Sebe, A. Garg, L. Chen, and T. Huang, "Facial expression recognition from video sequences: Temporal and static modeling," Computer Vision and Image Understanding 91(1-2), pp. 160-187, 2003.
13. L. Chen, Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction. PhD thesis, University of Illinois at Urbana-Champaign, 2000.
14. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T. Huang, "Semi-supervised learning of classifiers: Theory, algorithms, and applications to human-computer interaction," IEEE Trans. on Pattern Analysis and Machine Intelligence, to appear, 2004.
15. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey," Pattern Recognition 36, pp. 259-275, 2003.
16. M. Pantic and L. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. on Pattern Analysis and Machine Intelligence 22(12), pp. 1424-1445, 2000.
17. M. Pantic and L. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proceedings of the IEEE 91(9), pp. 1370-1390, 2003.
18. Y. Yacoob and L. Davis, "Recognizing human facial expressions from long image sequences using optical flow," IEEE Trans. on Pattern Analysis and Machine Intelligence 18(6), pp. 636-642, 1996.
19. J. Bassili, "Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face," Journal of Personality and Social Psychology 37(11), pp. 2049-2058, 1979.
20. I. Murray and J. Arnott, "Toward the simulation of emotion in synthetic speech: A review of the literature of human vocal emotion," Journal of the Acoustic Society of America 93(2), pp. 1097-1108, 1993.
21. C. Chiu, Y. Chang, and Y. Lai, "The analysis and recognition of human vocal emotions," in Proc. International Computer Symposium, pp. 83-88, 1994.
22. F. Dellaert, T. Polzin, and A. Waibel, "Recognizing emotion in speech," in Proc. International Conf. on Spoken Language Processing, pp. 1970-1973, 1996.
23. K. Scherer, "Adding the affective dimension: A new look in speech analysis and synthesis," in Proc. International Conf. on Spoken Language Processing, pp. 1808-1811, 1996.
24. Y. Sagisaka, N. Campbell, and N. Higuchi, eds., Computing Prosody, Springer-Verlag, New York, NY, 1997.
25. C. Williams and K. Stevens, "Emotions and speech: Some acoustical correlates," Journal of the Acoustic Society of America 52(4), pp. 1238-1250, 1972.
26. I. Murray and J. Arnott, "Synthesizing emotions in speech: Is it time to get excited?," in Proc. International Conf. on Spoken Language Processing, pp. 1816-1819, 1996.
27. V. Petrushin, "How well can people and computers recognize emotions in speech?," in Proc. AAAI Fall Symposium, pp. 141-145, 1998.
28. R. Picard, E. Vyzas, and J. Healey, "Toward machine emotional intelligence: Analysis of affective physiological state," IEEE Trans. on Pattern Analysis and Machine Intelligence 23(10), pp. 1175-1191, 2001.
29. J. Cacioppo and L. Tassinary, "Inferring psychological significance from physiological signals," American Psychologist 45, pp. 16-28, 1990.
30. W. James, The Principles of Psychology, Henry Holt, New York, NY, 1890.
31. W. Cannon, "The James-Lange theory of emotion: A critical examination and an alternative theory," American Journal of Psychology 39, pp. 106-124, 1927.
32. S. Schachter, "The interaction of cognitive and physiological determinants of emotional state," in Advances in Experimental Psychology, L. Berkowitz, ed., 1, pp. 49-80, 1964.
33. W. Winton, L. Putman, and R. Krauss, "Facial and autonomic manifestations of the dimensional structure of the emotion," Journal of Experimental Social Psychology 20, pp. 195-216, 1984.
34. A. Fridlund and C. Izard, "Electromyographic studies of facial expressions of emotions and patterns of emotion," in Social Psychophysiology: A Sourcebook, J. Cacioppo and R. Petty, eds., pp. 243-286, 1983.
35. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. Senior, "Recent advances in the automatic recognition of audiovisual speech," Proceedings of the IEEE 91(9), pp. 1306-1326, 2003.
36. C. Pelachaud, N. Badler, and M. Steedman, "Generating facial expression for speech," Cognitive Science 20, pp. 1-46, 1996.
37. L. De Silva and P. Ng, "Bimodal emotion recognition," in Proc. Automatic Face and Gesture Recognition, pp. 332-335, 2000.
38. Y. Medan, E. Yair, and D. Chazan, "Super resolution pitch determination of speech signals," IEEE Trans. on Signal Processing 39, pp. 40-48, 1991.
39. L. Chen and T. Huang, "Emotional expressions in audiovisual human computer interaction," in Proc. International Conference on Multimedia and Expo (ICME), pp. 423-426, 2000.
40. L. Chen, H. Tao, T. Huang, T. Miyasato, and R. Nakatsu, "Emotion recognition from audiovisual information," in Proc. IEEE Workshop on Multimedia Signal Processing, pp. 83-88, 1998.
41. H. Pan, Z. Liang, T. Anastasio, and T. Huang, "Exploiting the dependencies in information fusion," in Proc. Conf. on Computer Vision and Pattern Recognition, 2, pp. 407-412, 1999.
42. A. Garg, V. Pavlovic, and J. Rehg, "Boosted learning in dynamic Bayesian networks for multimodal speaker detection," Proceedings of the IEEE 91(9), pp. 1355-1369, 2003.
43. A. Pentland, "Looking at people," Communications of the ACM 43(3), pp. 35-44, 2000.
44. N. Sebe, M. Lew, I. Cohen, Y. Sun, T. Gevers, and T. Huang, "Authentic facial expression analysis," in Automatic Face and Gesture Recognition, pp. 517-522, 2004.
45. P. Ekman, ed., Emotion in the Human Face, Cambridge University Press, New York, NY, 2nd ed., 1982.
46. D. Goleman, Emotional Intelligence, Bantam Books, 1995.
47. B. Schneiderman, "Human values and the future of technology: A declaration of responsibility," in Sparks of Innovation in Human-computer Interaction, B. Schneiderman, ed., 1993.