Multimodal emotion recognition

Handbook of Pattern Recognition and Computer Vision, 3rd Edition

Volumn , Issue , 2005, Pages 387-410

  Sebe, Nicu ^a   Cohen, Ira ^b   Huang, Thomas S ^c  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b HEWLETT PACKARD LABORATORIES   (United States)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SPEECH RECOGNITION;

EMOTION RECOGNITION; FORCE FEEDBACK; HUMAN COMMUNICATIONS; MULTIMODAL EMOTION RECOGNITION; NATURAL INTERACTIONS; PROBABILISTIC GRAPHICAL MODELS; TECHNOLOGICAL ADVANCES; UNLABELED DATA;

HUMAN COMPUTER INTERACTION;

EID: 84968724649     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1142/9789812775320_0021     Document Type: Chapter

References (68)

1. J.N. Bassili. Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37(ll):2049-2058, 1979.
2. N. Bianchi-Berthouze and C. Lisetti. Modeling multimodal expression of user’s affective subjective experience. User Modeling and User-Adapted Interaction, 12:49-84, 2002.
3. M.J. Black and Y. Yacoob. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proc. International Conf. on Computer Vision, pages 374-381, 1995.
4. J.T. Cacioppo and L.G. Tassinary. Inferring psychological significance from physiological signals. American Psychologist, 45:16-28, 1990.
5. W.B. Cannon. The James-Lange theory of emotion: A critical examination and an alternative theory. American Journal of Psychology, 39:106-124, 1927.
6. L.S. Chen. Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction. PhD thesis, University of Illinois at Urbana-Champaign, Dept. of Electrical Engineering, 2000.
7. L.S. Chen and T.S. Huang. Emotional expressions in audiovisual human computer interaction. In Proc. International Conference on Multimedia and Expo (ICME), pages 423-426, 2000.
8. L.S. Chen, H. Tao, T.S. Huang, T. Miyasato, and R. Nakatsu. Emotion recognition from audiovisual information. In Proc. IEEE Workshop on Multimedia Signal Processing, pages 83-88, 1998.
9. C.C. Chiu, Y.L. Chang, and Y.J. Lai. The analysis and recognition of human vocal emotions. In Proc. International Computer Symposium, pages 83-88, 1994.
10. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang. Learning bayesian network classifiers for facial expression recognition using both labeled and unlabeled data. In Proc. Conf. on Computer Vision and Pattern Recognition, volume 1, pages 595-601, 2003.
11. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang. Semi-supervised learning of classifiers: Theory, algorithms, and applications to human-computer interaction. IEEE Trans, on Pattern Analysis and Machine Intelligence, to appear, 2004.
12. I. Cohen, N. Sebe, A. Garg, L. Chen, and T.S. Huang. Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding, 91(1-2):160-187, 2003.
13. R. Cowie and E. Douglas-Cowie. Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In Proc. International Conf. on Spoken Language Processing, pages 1989-1992, 1996.
14. C. Darwin. The Expression of the Emotions in Man and Animals. ohn Murray, London, 2nd edition, 1890.
15. L.C. De Silva and P.C Ng. Bimodal emotion recognition. In Proc. Automatic Face and Gesture Recognition, pages 332-335, 2000.
16. F. Dellaert, T. Polzin, and A. Waibel. Recognizing emotion in speech. In Proc. International Conf. on Spoken Language Processing, pages 1970-1973, 1996.
17. P. Ekman, editor. Emotion in the Human Face. ambridge University Press, New York, NY, 2nd edition, 1982.
18. P. Ekman. Strong evidence for universals in facial expressions: A reply to Russell’s mistaken critique. Psychological Bulletin, 115(2):268-287, 1994.
19. P. Ekman and W.V. Friesen. Facial Action Coding System: Investigator’s Guide. Consulting Psychologists Press, 1978.
20. LA. Essa and A.P. Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans, on Pattern Analysis and Machine Intelligence, 19(7):757-763, 1997.
21. B. Fasel and J. Luettin. Automatic facial expression analysis: A survey. Pattern Recognition, 36:259-275, 2003.
22. A. Fridlund and C. Izard. Electromyographic studies of facial expressions of emotions and patterns of emotion. In J. Cacioppo and R. Petty, editors, Social Psychophysiology:A Sourcebook, pages 243-286, 1983.
23. A. Garg, M. Naphade, and T.S. Huang. Modeling video using input/output markov models with application to multi-modal event detection. In B. Furht, O. Marques, and B. Furht, editors, Handbook of Video Databases: Design and Applications, 2003.
24. A. Garg, V. Pavlovic, and J. Rehg. Boosted learning in dynamic Bayesian networks for multimodal speaker detection. Proceedings of the IEEE, 91 (9): 1355-1369, 2003.
25. D. Goleman. Emotional Intelligence. Bantam Books, 1995.
26. E. Hilgard, R.C. Atkinson, and R.L. Hilgard. Introduction to Psychology. arcourt Brace Jovanovich, New York, NY, 5th edition, 1971.
27. C.E. Izard. Innate and universal facial expressions: Evidence from developmental and cross-cultural research. Psychological Bulletin, 115(2):288-299, 1994.
28. W. James. The Principles of Psychology. Henry Holt, New York, NY, 1890.
29. J.M. Jenkins, K. Oatley, and N.L. Stein, editors. Human Emotions: A Reader. Blackwell Publishers, Maiden, MA, 1998.
30. T. Johnstone. Emotional speech elicited using computer games. In Proc. International Conf. on Spoken Language Processing, pages 1985-1988, 1996.
31. P. Lang. The emotion probe: Studies of motivation and attention. American Psychologist, 50(5):372-385, 1995.
32. A. Lanitis, C.J. Taylor, and T.F. Cootes. A unified approach to coding and interpreting face images. In Proc. International Conf. on Computer Vision, pages 368-373, 1995.
33. J. Lien. Automatic recognition of facial expressions using hidden Markov models and estimation of expression intensity. PhD thesis, Carnegie Mellon University, 1998.
34. A Martinez. Face image retrieval using HMMs. In IEEE Workshop on Content-based Access of Images and Video Libraries, pages 35-39, 1999.
35. K. Mase. Recognition of facial expression from optical flow. IEICE Trans., E74(10):3474-3483, 1991.
36. D. Matsumoto. Cultural influences on judgments of facial expressions of emotion. In Proc. ATR Symposium on Face and Object Recognition, pages 13-15, 1998.
37. Y. Medan, E. Yair, and D. Chazan. Super resolution pitch determination of speech signals. IEEE Trans, on Signal Processing, 39:40-48, 1991.
38. A. Mehrabian. Communication without words. Psychology Today, 2(4):53-56, 1968.
39. S. Morishima. Emotion model: A criterion for recognition, synthesis and compression of face and emotion. In Proc. Automatic Face and Gesture Recognition, pages 284-289, 1995.
40. I.R. Murray and J.L. Arnott. Toward the simulation of emotion in synthetic speech:A review of the literature of human vocal emotion. Journal of the Acoustic Society of America, 93(2): 1097-1108, 1993.
41. I.R. Murray and J.L. Arnott. Synthesizing emotions in speech: Is it time to get excited? In Proc. International Conf. on Spoken Language Processing, pages 1816-1819, 1996.
42. A. Nefian and M. Hayes. Face recognition using an embedded HMM. In IEEE Conf. on Audio and Video-based Biometric Person Authentication, pages 19-24, 1999.
43. N. Oliver, A. Pentland, and F. Berard. LAFTER: A real-time face and lips tracker with facial expression recognition. Pattern Recognition, 33:1369-1382, 2000.
44. T. Otsuka and J. Ohya. Recognizing multiple persons’ facial expressions using HMM based on automatic extraction of significant frames from image sequences. In Proc. International Conf. on Image Processing, pages 546-549, 1997.
45. T. Otsuka and J. Ohya. A study of transformation of facial expressions based on expression recognition from temporal image sequences. Technical report, Institute of Electronic, Information, and Communications Engineers (IEICE), 1997.
46. H. Pan, Z.P. Liang, T.J. Anastasio, and T.S. Huang. Exploiting the dependencies in information fusion. In Proc. Conf. on Computer Vision and Pattern Recognition, volume 2, pages 407-412, 1999.
47. M. Pantic and L.J.M. Rothkrantz. Automatic analysis of facial expressions: The state of the art. IEEE Trans, on Pattern Analysis and Machine Intelligence, 22(12):1424-1445, 2000.
48. M. Pantic and L.J.M. Rothkrantz. Toward an affect-sensitive multimodal humancomputer interaction. Proceedings of the IEEE, 91 (9).-1370-1390, 2003.
49. C. Pelachaud, N. Badler, and M. Steedman. Generating facial expression for speech. Cognitive Science, 20:1-46, 1996.
50. A. Pentland. Looking at people. Communications of the ACM, 43(3):35-44, 2000.
51. V.A. Petrushin. How well can people and computers recognize emotions in speech? In Proc. AAAI Fall Symposium, pages 141-145, 1998.
52. R W. Picard. Affective Computing. MIT Press, Cambridge, MA, 1997.
53. R.W. Picard, E. Vyzas, and J. Healey. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans, on Pattern Analysis and Machine Intelligence, 23(10):1175-1191, 2001.
54. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A.W. Senior. Recent advances in the automatic recognition of audiovisual speech. Proceedings of the IEEE, 91(9):1306-1326, 2003.
55. B. Reeves and C. Nass. The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places. Cambridge Univ. Press, 1996.
56. M. Rosenblum, Y. Yacoob, and L.S. Davis. Human expression recognition from motion using a radial basis function network architecture. IEEE Trans, on Neural Network, 7(5):1121-1138, 1996.
57. Y. Sagisaka, N. Campbell, and N. Higuchi, editors. Computing Prosody. Springer-Verlag, New York, NY, 1997.
58. P. Salovey and J.D. Mayer. Emotional intelligence. Imagination, Cognition, and Personality, 9(3):185-211, 1990.
59. J. Sato and S. Morishima. Emotion modeling in speech production using emotion space. In Proc. IEEE Int. Workshop on Robot and Human Communication, pages 472-477, 1996.
60. S. Schachter. The interaction of cognitive and physiological determinants of emotional state. In L. Berkowitz, editor, Advances in Experimental Psychology, volume 1, pages 49-80, 1964.
61. K.R. Scherer. Adding the affective dimension: A new look in speech analysis and synthesis. In Proc. International Conf. on Spoken Language Processing, pages 1808-1811, 1996.
62. H. Schlosberg. Three dimensions of emotion. Psychological Review, 61:81-88, 1954.
63. B. Schneiderman. Human values and the future of technology: A declaration of responsibility. In B. Schneiderman, editor, Sparks of Innovation in Human-computer Interaction, 1993.
64. N. Sebe, M.S. Lew, I. Cohen, Y. Sun, T. Gevers, and T.S. Huang. Authentic facial expression analysis. In Automatic Face and Gesture Recognition, pages 517-522, 2004.
65. N. Ueki, S. Morishima, H. Yamada, and H. Harashima. Expression analysis/synthesis system based on emotion space constructed by multilayered neural network. Systems and Computers in Japan, 25(13):95-103, 1994.
66. C.E. Williams and K.N. Stevens. Emotions and speech: Some acoustical correlates. Journal of the Acoustic Society of America, 52(4):1238-1250, 1972.
67. W. Winton, L. Putman, and R. Krauss. Facial and autonomic manifestations of the dimensional structure of the emotion. Journal of Experimental Social Psychology, 20:195-216, 1984.
68. Y. Yacoob and L.S. Davis. Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans, on Pattern Analysis and Machine Intelligence, 18(6):636-642, 1996.