Toward an affect-sensitive multimodal human-computer interaction

Proceedings of the IEEE

Volumn 91, Issue 9, 2003, Pages 1370-1390

  Pantic, Maja ^a   Rothkrantz, Leon J M ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Affective computing; Affective states; Automatic analysis of nonverbal communicative cues; Human computer interaction (HCI); Multimodal human computer interaction; Personalized human computer interaction

Indexed keywords

ANIMATION; COMPUTATIONAL METHODS; FEEDBACK; INFORMATION TECHNOLOGY; LEARNING SYSTEMS; PATTERN RECOGNITION; SENSORY PERCEPTION; SIGNAL PROCESSING; SPEECH SYNTHESIS; USER INTERFACES;

AFFECTIVE COMPUTING; AFFECTIVE STATES; AUTOMATIC ANALYSIS OF NONVERBAL COMMUNICATIVE CUES; EMOTIONAL INTELLIGENCE; MULTIMODAL HUMAN-COMPUTER INTERACTION; PERSONALIZED HUMAN-COMPUTER INTERACTION; SENSORY SYSTEMS;

HUMAN COMPUTER INTERACTION;

EID: 2942590310     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2003.817122     Document Type: Conference Paper

References (126)

1. J. Y. Aloimonos, I. Weiss, and A. Bandyopadhyay, "Active vision," Int. J. Comput. Vis., vol. I, no. 4, pp. 333-356, 1988.
2. N. Amir and S. Ron, "Toward automatic classification of emotions in speech," in Proc. ICSLP, 1998, pp. 555-558.
3. P. Anandan, "A computational framework and an algorithm for the measurement of visual motion," Int. J. Comput. Vis., vol. 2, no. 3, pp. 283-310, 1989.
4. J. R. Averill, "Acquisition of emotions in adulthood," in The Social Construction of Emotions, R. Harre, Ed. Oxford, U.K.: Blackwell, 1986, p. 100.
5. R. Banse and K. R. Scherer, "Acoustic profiles in vocal emotion expression,"J. Personality Social Psychol., vol. 70, no. 3, pp. 614-636, 1996.
6. M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Measuring facial expressions by computer image analysis," Psychophysiology, vol. 36, pp. 253-263, 1999.
7. J. N. Bassili, "Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face," J. Personality Social Psychol., vol. 37, pp. 2049-2058, 1979.
8. R. V. Bezooijen, Characteristics and Recognizability of Vocal Expression of Emotions. Dordrecht, The Netherlands: Floris, 1984.
9. M. J. Black and Y. Yacoob, "Recognizing facial expressions in image sequences using local parameterized models of image motion," Int. J. Comput. Vis., vol. 25, no. 1, pp. 23-48, 1997.
10. E. Boyle, A. H. Anderson, and A. Newlands, "The effects of visibility on dialogue in a cooperative problem solving task," Lang. Speech, vol. 37, no. I. pp. 1-20, 1994.
11. V. Bruce, "What the human face tells the human mind: Some challenges for the robot-human interface," in Proc. ROMAN, 1992, pp. 44-51.
12. J. T. Cacioppo, G. G. Berntson, J. T. Larsen, K. M. Poehlmann, and T. A. Ito, "The psychophysiology of emotion," in Handbook of Emotions, M. Lewis and J. M. Havil-Jones, Eds. New York: Guilford, 2000, pp. 173-191.
13. F. W. Campbell and D. G. Green, "Optical and retinal factors affecting visual resolution," J. Physiol., vol. 181, pp. 576-593, 1965.
14. J. Cassell and T. Bickmore, "External manifestations of trust-worthiness in the interface," Commun. ACM, vol. 43, no. 12, pp. 50-56, 2000.
15. L. S. Chen and T. S. Huang, "Emotional expressions in audiovisual human computer interaction," in Proc. ICME, 2000, pp. 423-426.
16. (16] L. S. Chen, T. S. Huang, T. Miyasato, and R. Nakatsu, "Multimodal human emotion/expression recognition," in Proc. FG, 1998, pp. 396-401.
17. T. Chen and R. R. Rao, "Audio-visual integration in multimodal communication," Proc. IEEE, vol. 86, pp. 837-852, May 1998.
18. T. Chen, "Audiovisual speech processing," IEEE Signal Processing Mag., vol. 18, pp. 9-21, Jan. 2001.
19. I. Cohen, N. Sebe, A. Garg, M. S. Lew, and T. S. Huang, "Facial expression recognition from video sequences," in Proc. ICME. 2002, pp. 121-124.
20. "Special section on video surveillance," IEEE Trans. Pattern Anal. Machine Intell., vol. 22, pp. 745-887, Aug. 2000.
21. A. Colmenarez, B. Frey, and T. S. Huang, "Embedded face and facial expression recognition," in Proc. ICIP, vol. 1, 1999. pp. 633-637.
22. R. Cornelius, The Science of Emotion. Englewood Cliffs, NJ: Prentice-Hall, 1996.
23. G. W. Cottrell and J. Metcalfe, "EMPATH: Face, emotion, and gender recognition using holons," Adv. Neural Inf. Process. Syst., vol. 3, pp. 564-571, 1991.
24. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Processing Mag., vol. 18, pp. 32-80, Jan. 2001.
25. C. Darwin, The Expression of the Emotions in Man and Animals. Chicago, IL: Univ. of Chicago Press, 1965.
26. B. V. Dasarathy, "Sensor fusion potential exploitation - innovative architectures and illustrative approaches," Proc. IEEE. vol. 85, pp. 24-38, Jan. 1997.
27. J. R. Davitz, Ed., The Communication of Emotional Meaning. New York: McGraw-Hill, 1964.
28. L. C. De Silva, T. Miyasato, and R. Nakatsu, "Facial emotion recognition using multimodal information," in Proc. ICSP, 1997, pp. 397-401.
29. L. C. De Silva and P. C. Ng, "Bimodal emotion recognition," in Proc. FG, 2000, pp. 332-335.
30. F. Dellaert, T. Polzin, and A. Waibel, "Recognizing emotion in speech," in Proc. ICSLP, 1996, pp. 1970-1973.
31. G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Classifying facial actions," IEEE Trans. Pattern Anal. Machine Intell., vol. 21, pp. 974-989, Oct. 1999.
32. H. Ebine, Y. Shiga, M. Ikeda, and O. Nakamura, "The recognition of facial expressions with automatic detection of reference face," in Proc. Canadian Conf. ECE, vol. 2, 2000, pp. 1091-1099.
33. G. J. Edwards, T. F. Cooles, and C. J. Taylor, "Face recognition using active appearance models," in Proc. ECCV, vol. 2, 1998, pp. 581-695.
34. P. Ekman, "Universals and cultural differences in facial expressions of emotion," in Proc. Nebraska Symp. Motivation, J. Cole, Ed., 1972, pp. 207-283.
35. P. Ekman and W. F. Friesen, "The repertoire of nonverbal behavioral categories - origins, usage, and coding," Semiotica, vol. 1, pp. 49-98, 1969.
36. P. Ekman and W. Friesen, Facial Action Coding System. Palo Alto. CA: Consulting Psychologist, 1978.
37. P. Ekman, T. S. Huang, T. J. Sejnowski, and J. C. Hager, Eds., "Final report to NSF of the planning workshop on facial expression understanding," Human Interaction Lab., Univ. California, San Francisco, 1993.
38. P. Ekman, J. Hager, C. H. Methvin, and W. Irwin, "Ekman-Hager facial action exemplars," Human Interaction Lab., Univ. California, San Francisco.
39. I. Essa and A. Pentland, "Coding analysis interpretation recognition of facial expressions," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, pp. 757-763, July 1997.
40. W. A. Felienz, J. G. Taylor, R. Cowie, E. Douglas-Cowie, F. Piat, S. Kollias, C. Orovas, and B. Apolloni, "On emotion recognition of faces and of speech using neural networks, fuzzy logic and the assess system," in Proc. IJCNN, vol. 2, 2000, pp. 93-98.
41. I. Fonagy and K. Magdics, "Emotional patterns in intonation and music," Phonet, Sprachwiss. Kommunikutionsforsch, vol. 16, pp. 293-326, 1963.
42. R. Frick, "Communicating emotion. The role of prosodic features," Psychol. Bull., vol. 97, no. 3, pp. 412-429, 1985.
43. G. Furnas, T. Landauer, L. Gomes, and S. Dumais, "The vocabulary problem in human-system communication," Commun. ACM. vol. 30, no. 11, pp. 964-972, 1987.
44. D. Goleman, Emotional Intelligence. New York: Bantam Books, 1995.
45. D. L. Hall and J. Llinas, "An introduction to multisensor data fusion," Proc. IEEE. vol. 85, pp. 6-23, Jan. 1997.
46. A. Hanjalic and L.-Q. Xu, "User-oriented affective video content analysis," in Proc. IEEE Workshop Content-Based Access of Image and Video Libraries , 2001, pp. 50-57.
47. F. Hara and H. Kobayashi, "Facial interaction between animated 3D face robot and human beings," in Proc. SMC, 1997, pp. 3732-3737.
48. H. Hong, H. Neven, and C. von der Malsburg, "Online facial expression recognition based on personalised galleries," in Proc. FG, 1998, pp. 354-359.
49. C. L. Huang and Y. M. Huang, "Facial expression recognition using model-based feature extraction and action parameters classification," Vis. Commun. Image Representat., vol. 8, no. 3, pp. 278-290, 1997.
50. R. Huber, A. Batliner, J. Buckow, E. Moth, V. Warnke, and H. Niemann, "Recognition of emotion in a realistic dialogue scenario," in Proc. ICSLP, 2000, pp. 665-668.
51. G. Izzo. (1998) Review of Existing techniques for human emotion understanding and applications in HCI. [Online]. Available: http://www.image. ece.ntua.gr/physta/reports/emotionreview.
52. T. S. Jaakkola and D. Haussler, "Exploiting generative models in discriminative classifiers," in Proc. Advances in Neural Information Processing, vol. 11, M. S. Kearns, S. A. Solla, and D. A. Cohn, Eds., 1998, pp. 487-493.
53. S. Ji, C. Yoon, J. Park, and M. Park, "Intelligent system for automatic adjustment of 3D facial shape model and face expression recognition," in Proc. IFSC, vol. 3, 1999, pp. 1579-1584.
54. |54] "Special issue on spoken language processing," Proc. IEEE, vol. 88, pp. 1139-1366, Aug. 2000.
55. T. Kanade J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. FG, 2000, pp. 46-53.
56. B. S. Kang, C. H. Han, S. T. Lee, D. H. Youn, and C. Lee, "Speaker dependent emotion recognition using speech signals," in Proc. ICSLP, 2000, pp. 383-386.
57. G. D. Kearney and S. McKenzie, "Machine interpretation of emotion: Memory-based expert system for interpreting facial expressions in terms of signaled emotions (JANUS)," Cogn. Sci., vol. 17, no. 4, pp. 589-622, 1993.
58. D. Keltner and P. Ekman, "Facial expression of emotion," in Handhook of Emotions, M. Lewis and J. M. Havil-Jones, Eds. New York: Guilford, 2000, pp. 236-249.
59. [591 S. Kimura and M. Yachida, "Facial expression recognition and its degree estimation," in Proc. CVPR, 1997, pp. 295-300.
60. H. Kobayashi and F. Hara, "Recognition of mixed facial expressions by a neural network," in Proc. ROMAN, 1992, pp. 387-391 (see also pp. 381-386).
61. T. Kurozumi, Y. Shinza, Y. Kenmochi, and K. Kotani, "Facial individuality and expression analysis by eigenface method based on class features or multiple discriminant analysis," in Proc. ICIP, vol. 1, 1999, pp. 648-652.
62. Y. Li and Y. Zhao, "Recognizing emotions in speech using short-term and long-term features," in Proc. ICSLP, 1998, pp. 2255-2258.
63. [631 M. J. Lyons, J. Budynek, and S. Akamalsu, "Automatic classification of single facial images," IEEE Trans. Pattern Anal. Machine Intell., vol. 21, pp. 1357-1362, Dec. 1999.
64. S. Mann, "Wearable computing: A first step toward personal imaging," Computer, vol. 30, no. 2, pp. 25-32, 1997.
65. I. Marsic, A. Medl, and J. Flanagan, "Natural communication with information systems," Proc. IEEE, vol. 88, pp. 1354-1366, Aug. 2000.
66. K. Mase, "Recognition of facial expression from optical How," IEICE Trans., vol. E74, no. 10, pp. 3474-3483, 1991.
67. D. Malsumoto, "Cultural similarities and differences in display rules," Motivat. Emotion, vol. 14, pp. 195-214, 1990.
68. D. McNeill, Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
69. [69j G. McRoberts, M. Studdert-Kennedy, and D. P. Shankweiler, "Role of fundamental frequency in signaling linguistic stress and affect: Evidence for a dissociation," Percept. Psychophys., vol. 57, pp. 159-174, 1995.
70. Y. Medan, E. Yair, and D. Chazan, "Super resolution pitch determination of speech signals," IEEE Trans. Signal Processing, vol. 39, pp. 40-48, Jan. 1991.
71. A. Mehrabian, "Communication without words," Psychol. Today, vol. 2, no. 4, pp. 53-56, 1968.
72. I. R. Murray and J. L. Arnott, "Synthesizing emotions in speech: Is it time to get excited?," in Proc. ICSLP, 1996, pp. 1816-1819.
73. R. Nakatsu, "Toward the creation of a new medium for the multimedia era," Proc. IEEE, vol. 86, pp. 825-836, May 1998.
74. R. Nakatsu, J. Nicholson, and N. Tosa, "Emotion recognition and its application to computer agents with spontaneous interactive capabilities," Knowl.-Based Syst., vol. 13, no. 7-8, pp. 497-504, 2000.
75. C. I. Nass, J. S. Steuer, and E. Tauber, "Computers are social actors," in Proc. CHI, 1994, pp. 72-78.
76. T. L. Nwe, F. S. Wei, and L. C. De Silva, "Speech-based emotion classification," in Proc. TENCON. vol. 1, 2001, pp. 297-301.
77. I. Oakley, M. R. McGee, S. Brewster, and P. Grey, "Putting the feel in "look and feel'," in Proc. CHI , 2000, pp. 415-422.
78. J. S. Olson and G.M. Olson, "Trust in e-commerce," Commun. ACM, vol. 43, no. 12, pp. 41-44, 2000.
79. A. Ortony and T. J. Turner, "What is basic about emotions?," Psychol. Rev., vol. 74, pp. 315-341, 1990.
80. T. Otsuka and J. Ohya, "Spotting segments displaying facial expression from image sequences using HMM," in Proc. FG. 1998, pp. 442-447.
81. S. Oviatt, A. DeAngeli, and K. Kuhn, "Integration-synchronization of input modes during multimodal human-computer interaction," in Proc. CHI, 1997, pp. 415-422.
82. C. Padgett and G. W. Cottrell, "Representing face images for emotion classification," in Proc. Advances in Neural Information Processing Systems, 1996, pp. 894-900.
83. H. Pan, Z. P. Liang, T. J. Anastasio, and T. S. Huang, "Exploiting the dependencies in information fusion," Proc CVPR. vol. 2, pp. 407-412, 1999.
84. M. Pantic and L. J. M. Rothkrantz, "Expert system for automatic analysis of facial expression," Image Vis. Comput, J., vol. 18, no. 11, pp. 881-905, 2000.
85. _, "Automatic analysis of facial expression: The state of the art," IEEE Trans. Pattern Anal. Machine Intell., vol. 22, pp. 1424-1445, Dec. 2000.
86. _, "Affect-sensitive multi-modal monitoring in ubiquitous computing: Advances and challenges," in Proc. ICEIS, 2001, pp. 466-474.
87. M. Pantic, "Facial expression analysis by computational intelligence techniques," Ph.D. dissertation, Delft Univ. Technol., Delft, The Netherlands, 2001.
88. A. Pentland, "Looking at people: Sensing for ubiquitous and wearable computing," IEEE Trans. Pattern Anal Machine Intell., vol. 22, pp. 107-119, Jan. 2000.
89. V. A. Petrushin, "Emotion recognition in speech signal," in Proc. ICSLP, 2000, pp. 222-225.
90. (90] R. W. Picard, Affective Computing. Cambridge, MA: MIT Press, 1997.
91. R. W. Picard, E. Vyzas, and J. Healey, "Toward machine emotional intelligence: Analysis of affective physiological state," IEEE Trans. Pattern Anal. Machine Intell., vol. 23, pp. 1175-1191, Oct. 2001.
92. T. S. Polzin, "Detecting verbal and non-verbal cues in the communications of emotions," Ph.D. dissertation. School Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA. 2000.
93. B. Reeves and C. I. Nass, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. New York: Cambridge Univ. Press, 1996.
94. J. A. Russell, "Is there universal recognition of emotion from facial expression?," Psychol. Bull., vol. 115, no. 1, pp. 102-141, 1994.
95. J. A. Russell and J. M. Fernez-Dols, Eds., The Psychology of Facial Expression. Cambridge, U.K.: Cambridge Univ. Press, 1997.
96. P. Salovey and J. D. Mayer, "Emotional intelligence," Imaginat., Cogn. Personality, vol. 9, no. 3, pp. 185-211, 1990.
97. A. Samal and P. A. Iyengar, "Automatic recogniion and analysis of human faces and facial expressions: A survey," Pattern Recognit., vol. 25, no. 1, pp. 65-77, 1992.
98. H. Sato, Y. Mitsukura, M. Fukumi, and N. Akamatsu, "Emotional speech classification with prosodic parameters by using neural networks," in Proc. Australian and New Zealand Intelligent Information Systems Conf., 2001, pp. 395-398.
99. M. Satyanarayanan, "Pervasive computing: Vision and challenges," IEEE Pers. Commun., vol. 8, pp. 10-17, Aug. 2001.
100. P. Scanlon and R. B. Reilly, "Feature analysis or automatic speech reading," in Proc. IEEE Int. Workshop Multimedia Signal Processing, 2001, pp. 625-630.
101. D. J. Schiano, S. M. Ehrlich, K. Rahardja, and K. Sheridan, "Face to interface: Facial affect in human and machine," in Proc. CHI, 2000, pp. 193-200.
102. K. Scherer, T. Johnstone, and T. Banziger, "Automatic verification of emotionally stressed speakers: The problem of individual differences," in Proc. SPECOM, 1998, pp. 233-238.
103. B. Schneiderman, "Human values and the future of technology: A declaration of responsibility," in Sparks of Innovation in Human-Computer Interaction, B. Schneiderman, Ed. Norwood, NJ: Ablex, 1993.
104. N. Sebe, M. S. Lew, I. Cohen, A. Garg, and T. S. Huang, "Emotion recognition using a cauchy naive bayes classifier," in Proc. ICPR, vol. 1, 2002. pp. 17-20.
105. R. Sharma, V. I. Pavlovic, and T. S. Huang, "Toward multimodal human-computer interface," Proc. IEEE. vol. 86, pp. 853-869, May 1998.
106. S. Shigeno, "Cultural similarities and differences in recognition of audio-visual speech stimuli," in Proc. ICSLP, 1998, pp. 281-284.
107. H. J. M. Steeneken and J. H. L. Hansen, "Speech under stress conditions: Overview of the effect on speech production and on system performance," in Proc. ICASSP, vol. 4, 1999, pp. 2079-2082.
108. B. Stein and M. A. Meredith, The Merging of Senses. Cambridge, MA: MIT Press, 1993.
109. J. Sulc, "Emotional changes in human voice," Activitas Nervosa Superior, vol. 19, pp. 215-216, 1977.
110. A. Takeuchi and K. Nagao, "Communicative facial displays as a new conversational modality," in Proc. INTERCHI, 1993, pp. 187-193.
111. H. Tao and T. S. Huang, "Explanation-based facial motion tracking using a piecewise bezier volume deformation model," in Proc. CVPR, vol. 1, 1999, pp. 611-617.
112. Y. Tian, T. Kanade, and J. F. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Machine Intell., vol. 23, pp. 97-115, Feb. 2001.
113. N. Tosa and R. Nakatsu, "Life-like communication agent - emotion sensing character MIC and feeling session character MUSE," in Proc. Int. Conf. Multimedia Computing and Systems, 1996, pp. 12-19.
114. V. Vapnik, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1999.
115. E. Viennet and F. F. Soulie, "Multi-resolufion scene segmentation by MLP's," in Proc. IJCNN, vol. 3, 1992, pp. 55-59.
116. M. Wang, Y. Iwai, and M. Yachida, "Expression recognition from time-sequential facial images by use of expression change model," in Proc. FG, 1998, pp. 324-329.
117. A. Wierzbicka, "Reading human faces," Pragmat. Cogn., vol. 1, no. 1, pp. 1-23, 1993.
118. C. E. Williams and K. N. Stevens, "Emotions and speech: Some acoustic correlates," J. Acoust. Soc. Amer, vol. 52, pp. 1238-1250, 1972.
119. Y. Yacoob and L. Davis, "Recognizing facial expressions by spatio-temporal analysis," in Proc. ICPR, vol. 1, 1994, pp. 747-749.
120. M. H. Yang, D. J. Kriegman, and N. Ahuja, "Detecting faces in images: A survey," IEEE Trans. Pattern Anal. Machine Intell., vol. 24, pp. 34-58, Jan. 2002.
121. M. H. Yang, D. Roth, and N. Ahuja, "A SNoW-based face detector," in Proc. Neural Information Processing Systems, vol. 12, 2000, pp. 855-861.
122. Y. Yoshilomi, S. Kim, T. Kawano, and T. Kiiazoe, "Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face," in Proc. ROMAN, 2000, pp. 178-183.
123. Y. Yoshitomi, N. Miyawaki, S. Tomita, and S. Kimura, "Facial expression recognition using thermal image processing and neural network," in Proc. ROMAN, 1997, pp. 380-385.
124. Z. Zhang, M. Lyons, M. Schuster, and S. Akamatsu, "Comparison between geometry-based and gabor waveleis-based facial expression recognition using multi-layer perceptron," in Proc. FG, 1998, pp. 454-459.
125. L. Zhao, W. Lu, Y Jiang, and Z. Wu, "A siudy on emotional feature recognition in speech," in Proc. ICSLP, 2000, pp. 961-964.
126. Y. Zhu, L. C. De Silva, and C. C. Ko, "Using moment invariants and HMM in facial expression recognition," Pattern Recognit. Lett., vol. 23, no. 1-3, pp. 83-91, 2002.