Multimodal human-computer interaction: A survey

Computer Vision and Image Understanding

Volumn 108, Issue 1-2, 2007, Pages 116-134

  Jaimes, Alejandro ^a   Sebe, Nicu ^b  

^a IDIAP RESEARCH INSTITUTE   (Switzerland)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Affective computing; Gesture recognition; Human centered computing; Multimodal human computer interaction

Indexed keywords

COMPUTER VISION; FACE RECOGNITION; GESTURE RECOGNITION; MATHEMATICAL MODELS; USER INTERFACES;

AFFECTIVE COMPUTING; HUMAN CENTERED COMPUTING; MULTIMODAL HUMAN COMPUTER INTERACTION; TASK MODELING;

HUMAN COMPUTER INTERACTION;

EID: 34548206204     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2006.10.019     Document Type: Article

References (193)

1. Adjoudani A., and Benoit C. On the integration of auditory and visual parameters in an HMM-based ASR. In: Stork D., and Hennecke M. (Eds). Speech Reading by Humans and Machines (1996), Springer, Berlin, NY
2. Adler A., Eisenstein J., Oltmans M., Guttentag L., and Davis R. Building the design studio of the future. AAAI Fall Symposium on Making Pen-Based Interaction Intelligent and Natural (2004)
3. Aggarwal J.K., and Cai Q. Human motion analysis: a review. CVIU 73 3 (1999) 428-440
4. Ambady N., and Rosenthal R. Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychological Bulletin 111 2 (1992) 256-274
5. Arts E. Ambient intelligence: a multimedia perspective. IEEE Multimedia 11 1 (2004) 12-19
6. Balabanovic M. Exploring versus exploiting when learning user models for text recommendations. User Modeling and User-adapted Interaction 8 (1998) 71-102
7. Balomenos T., Raouzaiou A., Ioannou S., Drosopoulos S., Karpouzis A., and Kollias S. Emotion analysis in man-machine interaction systems. Workshop on Machine Learning for Multimodal Interaction (2005)
8. Ben-Arie J., Wang Z., Pandit P., and Rajaram S. Human activity recognition using multidimensional indexing. IEEE Transactions on PAMI 24 8 (2002) 1091-1104
9. Benali-Khoudja M., Hafez M., Alexandre J.-M., and Kheddar A. Tactile interfaces: a state-of-the-art survey. International Symposium on Robotics (2004)
10. Bernsen N.O. A reference model for output information in intelligent multimedia presentation systems. European Conference on Artificial Intelligence (1996)
11. Bernsen N.O. Foundations of multimodal representations: a taxonomy of representational modalities. Interacting with Computers 6 4 (1994) 347-371
12. Bernsen N.O. Defining a taxonomy of output modalities from an HCI perspective. Computer Standards and Interfaces, Special Double Issue 18 6-7 (1997) 537-553
13. Bernsen N.O. Multimodality in language and speech systems-from theory to design support tool. In: Granström B. (Ed). Multimodality in Language and Speech Systems (2001), Kluwer Academic Publishers, Dordrecht
14. Bianchi-Berthouze N., and Lisetti C. Modeling multimodal expression of user's affective subjective experience. User Modeling and User-adapted Interaction 12 (2002) 49-84
15. Black M., and Yacoob Y. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. International Conference on Computer Vision (1995) 374-381
16. Blattner M., and Glinert E. Multimodal integration. IEEE Multimedia 3 4 (1996) 14-24
17. Bobick A.F., and Davis J. The recognition of human movement using temporal templates. IEEE Transaction on PAMI 23 3 (2001) 257-267
18. Bolt R. Put-That-There: voice and gesture at the graphics interface. Computer Graphics 14 3 (1980) 262-270
19. Borst C., and Volz R. Evaluation of a haptic mixed reality system for interactions with a virtual control panel. Presence: Teleoperators and Virtual Environments 14 6 (2005)
20. Bradbury J.S., Shell J.S., and Knowles C.B. Hands on cooking: towards an attentive kitchen. ACM Conference Human Factors in Computing Systems (CHI) (2003)
21. Bregler C., and Konig Y. Eigenlips for robust speech recognition. International Conference on Acoustics, Speech, and Signal Processing (1994)
22. Brewster S.A., Lumsden J., Bell M., Hall M., and Tasker S. Multimodal 'Eyes-Free' interaction techniques for wearable devices. ACM Conference Human Factors in Computer Systems (CHI) (2003)
23. Campbell C.S., and Maglio P.P. A robust algorithm for reading detection. ACM Workshop on Perceptive User Interfaces (2001)
24. Card S.K., Moran T., and Newell A. The Psychology of Human-Computer Interaction (1983), Lawrence Erlbaum Associates, London
25. Carpenter R. The Logic of Typed Feature Structures (1992), Cambridge University Press
26. L.S. Chen, Joint Processing of Audio-visual Information for the Recognition of Emotional Expressions in Human-computer Interaction, PhD thesis, Univ. of Illinois at Urbana-Champaign (2000).
27. Chen L.S., Travis Rose R., Parrill F., Han X., Tu J., Huang Z., Harper M., Quek F., McNeill D., Tuttle R., and Huang T.S. VACE multimodal meeting corpus. MLMI (2005)
28. Chen D., Malkin R., and Yang J. Multimodal detection of human interaction events in a nursing home environment. Conference on Multimodal Interfaces (ICMI) (2004)
29. Cheyer A., and Julia L. MVIEWS: multimodal tools for the video analyst. Conference on Intelligent User Interfaces (IUI) (1998)
30. Cohen I., Sebe N., Cozman F., Cirelo M., and Huang T.S. Semi-supervised learning of classifiers: theory, algorithms, and their applications to human-computer interaction. IEEE Transactions on PAMI 22 12 (2004) 1553-1567
31. Cohen I., Sebe N., Garg A., Chen L., and Huang T.S. Facial expression recognition from video sequences: temporal and static modeling. CVIU 91 1-2 (2003) 160-187
32. Cohen P.R., Johnston M., McGee D.R., Oviatt S.L., Pittman J., Smith I., Chen L., and Clow J. QuickSet: multimodal interaction for distributed applications. ACM Multimedia (1997) 31-40
33. Cohen P.R., and McGee D.R. Tangible multimodal interfaces for safety-critical applications. Communications of the ACM 47 1 (2004) 1-46
34. Cohen P.R., and Oviatt S.L. The role of voice in human-machine communication. In: Roe D., and Wilpon J. (Eds). Voice Communication between Humans and Machines (1994), National Academy Press, Washington, DC
35. Computer Vision and Image Understanding, Special Issue on Eye Detection and Tracking 98 (1) (2005).
36. Corradini A., Mehta M., Bernsen N., and Martin J.-C. Multimodal input fusion in human-computer interaction. NATO-ASI Conference on Data Fusion for Situation Monitoring, Incident Detection, Alert, and Response Management (2003)
37. Dickie C., Vertegaal R., Fono D., Sohn C., Chen D., Cheng D., Shell J.S., and Aoudeh O. Augmenting and sharing memory with eye. Blog in CARPE (2004)
38. Dix A., Finlay J., Abowd G., and Beale R. Human-computer Interaction (2003), Prentice Hall, Englewood Cliffs, NJ
39. Duric Z., Gray W., Heishman R., Li F., Rosenfeld A., Schoelles M., Schunn C., and Wechsler H. Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. Proceedings of the IEEE 90 7 (2002) 1272-1289
40. Duchowski A.T. A breadth-first survey of eye tracking applications. Behavior Research Methods, Instruments, and Computing 34 4 (2002) 455-470
41. Dusan S., Gadbois G.J., and Flanagan J. Multimodal interaction on PDA's integrating speech and pen inputs. Eurospeech (2003)
42. In: Ekman P. (Ed). Emotion in the Human Face (1982), Cambridge University Press
43. Elting C., Rapp S., Mohler G., and Strube M. Architecture and implementation of multimodal plug and play. ICMI (2003)
44. Essa I., and Pentland A. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on PAMI 19 7 (1997)
45. Fagiani C., Betke M., and Gips J. Evaluation of tracking methods for human-computer interaction. IEEE Workshop on Applications in Computer Vision (2002)
46. Fasel B., and Luettin J. Automatic facial expression analysis: a survey. Pattern Recognition 36 (2003) 259-275
47. Fitzmaurice G., Ishii H., and Buxton W. Bricks: Laying the foundations for graspable user interfaces. ACM Conference Human Factors in Computer Systems (CHI) 1 (1995) 442-449
48. Flippo F., Krebs A., and Marsic I. A framework for rapid development of multimodal interfaces. ICMI (2003)
49. Fong T., Nourbakhsh I., and Dautenhahn K. A survey of socially interactive robots. Robotics and Autonomous Systems 42 3-4 (2003) 143-166
50. Fritz G., Seifert C., Luley P., Paletta L., and Almer A. Mobile vision for ambient learning in urban environments. International Conference on Mobile Learning (MLEARN) (2004)
51. Fussell S., Setlock L., Yang J., Ou J., Mauer E., and Kramer A. Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction 19 3 (2004) 273-309
52. Garg A., Naphade M., and Huang T.S. Modeling video using input/output Markov models with application to multi-modal event detection. Handbook of Video Databases: Design and Applications (2003)
53. Garg A., Pavlovic V., and Rehg J. Boosted learning in dynamic Bayesian networks for multimodal speaker detection. Proceedings of the IEEE 91 9 (2003) 1355-1369
54. Gatica-Perez D. Analyzing group interactions in conversations: a survey. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (2006) 41-46
55. Gavrila D.M. The visual analysis of human movement: a survey. CVIU 73 1 (1999) 82-98
56. Go H.J., Kwak K.C., Lee D.J., and Chun M. Emotion recognition from facial image and speech signal. Conference on the Society of Instrument and Control Engineers (2003)
57. Grauman K., Betke M., Lombardi J., Gips J., and Bradski G. Communication via eye blinks and eyebrow raises: video-based human-computer interfaces. Universal Access in the Information Society 2 4 (2003) 359-373
58. Gunes H., Piccardi M., and Jan T. Face and body gesture recognition for a vision-based multimodal analyzer. Pan-Sydney Area Workshop on Visual Information Processing 36 (2004)
59. Hakeem A., and Shah M. Ontology and taxonomy collaborated framework for meeting classification. ICPR (2004)
60. Hanjalic A., and Xu L.-Q. Affective video content representation and modeling. IEEE Transactions on Multimedia 7 1 (2005) 143-154
61. Heishman R., Duric Z., and Wechsler H. Using eye region biometrics to reveal affective and cognitive states. CVPR Workshop on Face Processing in Video (2004)
62. Hjelmas E., and Low B.K. Face detection: a survey. CVIU 83 (2001) 236-274
63. Hook K. Designing and evaluating intelligent user interfaces. International Conference on Intelligent User Interfaces (1999)
64. Hu W., Tan T., Wang L., and Maybank S. A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man, and Cybernetics 34 3 (2004)
65. IEEE Computer, Special Issue on Human-centered Computing, A. Jaimes, N. Sebe, D. Gatica-Perez, T.S. Huang (Eds.), 40 (5) (2007).
66. International Journal of Human-Computer Studies, Special Issue on Applications of Affective Computing in Human-computer Interaction 59(1-2) (2003).
67. Intille S., Larson K., Beaudin J., Nawyn J., Tapia E., and Kaushik P. A living laboratory for the design and evaluation of ubiquitous computing technologies. ACM Conference on Human Factors in Computing Systems (CHI) (2004)
68. Jacob R. The use of eye movements in human-computer interactions techniques: what you look at is what you get. ACM Transactions on Information Systems 9 3 (1991) 152-169
69. Jaimes A. Human-centered multimedia: culture, deployment, and access. IEEE Multimedia 13 1 (2006) 12-19
70. Jaimes A., and Liu J. Hotspot components for gesture-based interaction. IFIP Interact (2005)
71. Jaimes A., Nagamine T., Liu J., Omura K., and Sebe N. Affective meeting video analysis. IEEE International Conference on Multimedia and Expo (2005)
72. Jaimes A., and Sebe N. Multimodal human computer interaction: a survey. IEEE International Workshop on Human-Computer Interaction (2005)
73. Jaimes A., Sebe N., and Gatica-Perez D. Human-centered computing: a multimedia perspective. ACM Multimedia (2006)
74. Jain R. Folk computing. Communcation of the ACM 46 4 (2003) 27-29
75. Jameson A., Schafer R., Weis T., Berthold A., and Weyrath T. Making systems sensitive to the user's time and working memory constraints. International Conference on Intelligent User Interfaces (1997)
76. Ji Q., and Yang X. Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8 (2002) 357-377
77. Johnston M., and Bangalore S. Multimodal Applications from Mobile to Kiosk. W3C Workshop on Multimodal Interaction (2002)
78. El Kaliouby R., and Robinson P. Real time inference of complex mental states from facial expressions and head gestures. CVPR Workshop on Real-time Vision for HCI (2004)
79. Kay M. Functional unification grammar: a formalism for machine translation. International Conference on Computational Linguistics (1984)
80. Kettebekov S., and Sharma R. Understanding gestures in multimodal human computer interaction. International Journal on Artificial Intelligence Tools 9 2 (2000) 205-223
81. Kirishima T., Sato K., and Chihara K. Real-time gesture recognition by learning and selective control of visual interest points. IEEE Transactions on PAMI 27 3 (2005) 351-364
82. In: Kisacanin B., Pavlovic V., and Huang T.S. (Eds). Real-Time Vision for Human-Computer Interaction (2005), Springer-Verlag
83. Kono Y., Kawamura T., Ueoka T., Murata S., and Kidode M. Real world objects as media for augmenting human memory. Workshop on Multi-User and Ubiquitous User Interfaces(MU3I) (2004) 37-42
84. Kumar S., and Cohen P. Towards a fault-tolerant multi-agent system architecture. International Conference on Autonomous Agents (2000)
85. Kuno Y., Shimada N., and Shirai Y. Look where you're going: a robotic wheelchair based on the integration of human and environmental observations. IEEE Robotics and Automation 10 1 (2003) 26-34
86. Lang P. The emotion probe: studies of motivation and attention. American Psychologist 50 5 (1995) 372-385
87. Langley P. User modeling in adaptive interfaces. International Conference on User Modeling (1998)
88. Lanitis A., Taylor C.J., and Cootes T. A unified approach to coding and interpreting face images. International Conference on Computer Vision (1995) 368-373
89. Lauruska V., and Serafinavicius P. Smart home system for physically disabled persons with verbal communication difficulties. Assistive Technology Research Series (AAATE) (2003) 579-583
90. Legin A., Rudnitskaya A., Seleznev B., and Vlasov Y. Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie. Analytica Chimica Acta 534 (2005) 129-135
91. J. Lien, Automatic recognition of facial expressions using hidden Markov models and estimation of expression intensity, PhD thesis, Carnegie Mellon University, 1998.
92. Lyons M.J., Haehnel M., and Tetsutani N. Designing, playing, and performing, with a vision-based mouth Interface. Conference on New Interfaces for Musical Expression (2003)
93. Malkawi A.M., and Srinivasan R.S. Multimodal human-computer interaction for immersive visualization: integrating speech-gesture recognition and augmented reality for indoor environments. International Association of Science and Technology for Development Conference on Computer Graphics and Imaging (2004)
94. S. Marcel, Gestures for multi-modal interfaces: a review, Technical Report IDIAP-RR 02-34, 2002.
95. Martin D., Cheyer A., and Moran D. The open agent architecture: a framework for building distributed software systems. Applied Artificial Intelligence 13 (1999) 91-128
96. Martinez A. Face image retrieval using HMMs. IEEE workshop on content-based access of images and video libraries (1999) 35-39
97. Mase K. Recognition of facial expressions from optical flow. IEICE Transactions E74 10 (1991) 3474-3483
98. M. Maybury, Intelligent Multimedia Interfaces, AAAI/MIT Press, 1993.
99. Maynes-Aminzade D., Pausch R., and Seitz S. Techniques for interactive audience participation. ICMI (2002)
100. McCowan I., Gatica-Perez D., Bengio S., Lathoud G., Barnard M., and Zhang D. Automatic analysis of multimodal group actions in meetings. IEEE Transactions on PAMI 27 3 (2005) 305-317
101. McNeill D. Hand and Mind: What Gestures Reveal About Thought (1992), University of Chicago Press
102. Mehrabian A. Communication without words. Psychology Today 2 4 (1968) 53-56
103. Meyer S., and Rakotonirainy A. A Survey of research on context-aware homes. Australasian Information Security Workshop Conference on ACSW Frontiers (2003)
104. Minsky M. A framework for representing knowledge. In: Winston P. (Ed). The Psychology of Computer Vision (1975), McGraw-Hill, New York
105. Moeslund T.B., and Granum E. A survey of computer vision-based human motion capture. CVIU 81 3 (2001) 231-258
106. Moriyama T., Kanade T., Xiao J., and Cohn J. Meticulously detailed eye region model and its application to analysis of facial images. IEEE Transactions on PAMI 28 5 (2006) 738-752
107. Murray I.R., and Arnott J.L. Toward the simulation of emotion in synthetic speech: a review of the literature of human vocal emotion. Journal of the Acoustic Society of America 93 2 (1993) 1097-1108
108. Nefian A., and Hayes M. Face recognition using an embedded HMM. IEEE Conference on Audio and Video-based Biometric Person Authentication (1999) 19-24
109. Nielsen J. Non-command user interfaces. Communication of the ACM 36 4 (1993) 83-99
110. Nijholt A., and Heylen D. Multimodal communication in inhabited virtual environments. International Journal of Speech Technology 5 (2002) 343-354
111. D.A. Norman, The Design of Everyday Things, Doubleday, 1988.
112. Obrenovic Z., and Starcevic D. Modeling multimodal human-computer interaction. IEEE Computer (2004) 65-72
113. Oliver N., Pentland A., and Berard F. LAFTER: a real-time face and lips tracker with facial expression recognition. Pettern Recognition 33 (2000) 1369-1382
114. Otsuka T., and Ohya J. Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences. IEEE Conference on Image Processing (1997) 546-549
115. Otsuka T., and Ohya J. A study of transformation of facial expressions based on expression recognition from temporal image sequences. Technical Report, Institute of Electronic information, and Communications Engineers (IEICE) (1997)
116. Oudeyer P.Y. The production and recognition of emotions in speech: features and algorithms. International Journal of Human-Computer Studies 59 1-2 (2003) 157-183
117. Oulasvirta A., and Salovaara A. A cognitive meta-analysis of design approaches to interruptions in intelligent environments. ACM Conference Human Factors in Computing Systems (CHI) (2004)
118. Qvarfordt P., and Zhai S. Conversing with the user based on eye-gaze patterns. ACM Conference on Human Factors in Computing Systems (CHI) (2005)
119. Oviatt S.L., Lunsford R., and Coulston R. Individual differences in multimodal integration patterns: what are they and why do they exist?. ACM Conference Human Factors in Computing Systems (CHI) (2005)
120. Oviatt S.L. Mutual disambiguation of recognition errors in a multimodal architecture. ACM Conference Human Factors in Computing Systems (CHI) (1999)
121. Oviatt S.L. Ten myths of multimodal interaction. Communication of the ACM 42 11 (1999) 74-81
122. S.L. Oviatt, T. Darrell, M. Flickner (Eds.), Communications of the ACM, Special Issue on Multimodal interfaces that flex, adapt, and persist, 47(1), 2004.
123. Oviatt S.L., and Cohen P. Multimodal interfaces that process what comes naturally. Communications of the ACM 43 3 (2000) 45-48
124. S.L. Oviatt, Multimodal interfaces, Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, chap.14, 286-304, 2003.
125. Oviatt S.L., Cohen P., Wu L., Vergo J., Duncan L., Suhm B., Bers J., Holzman T., Winograd T., Landay J., Larson J., and Ferro D. Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Human-Computer Interaction 15 (2000) 263-322
126. Paggio P., and Jongejan B. Multimodal communication in the virtual farm of the staging Project. In: Stock O., and Zancanaro M. (Eds). Multimodal Intelligent Information Presentation (2005), Kluwer Academic Publishers, Dordrecht 27-46
127. Pan H., Liang Z.P., Anastasio T.J., and Huang T.S. Exploiting the dependencies in information fusion. CVPR 2 (1999) 407-412
128. Pantic M., and Rothkrantz L.J.M. Automatic analysis of facial expressions: the state of the art. IEEE Transactions on PAMI 22 12 (2000)
129. Pantic M., and Rothkrantz L.J.M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91 9 (2003) 1370-1390
130. Pantic M., Sebe N., Cohn J., and Huang T.S. Affective multimodal human-computer interaction. ACM Multimedia (2005)
131. J. Paradiso and F. Sparacino, Optical tracking for music and dance performance, Optical 3-D Measurement Techniques IV, A. Gruen, H. Kahmen, eds., pp. 11-18, 1997.
132. Pattern Recognition Letters, Special Issue on Multimodal Biometrics, vol. 24, No. 13, September 2003.
133. Pavlovic V.I., Sharma R., and Huang T.S. Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Transactions on PAMI 19 7 (1997) 677-695
134. Pelz J.B. Portable eye-tracking in natural behavior. Journal of Vision 4 11 (2004)
135. Pentland A. Looking at people. Communications of the ACM 43 3 (2000) 35-44
136. Pentland A. Socially aware computation and communication. IEEE Computer 38 3 (2005)
137. Picard R.W. Affective Computing (1997), MIT Press
138. Picard R.W., Vyzas E., and Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on PAMI 23 10 (2001) 1175-1191
139. Porta M. Vision-based user interfaces: methods and applications. International Journal on Human-Computer Studies 57 1 (2002) 27-73
140. Potamianos G., Neti C., Luettin J., and Matthews I. Audio-visual automatic speech recognition: an overview. In: Vatikiotis-Bateson E., and Perrier P. (Eds). Issues in Visual and Audio-Visual Speech Processing (2004), MIT Press
141. Proceedings of the IEEE, Special Issue on Multimodal Human Computer Interface. August, 2003.
142. Rabiner L., and Juang B. Fundamentals of Speech Recognition (1993), Prentice Hall, Englewood Cliffs, NJ
143. Reeves L.M., Martin J.-C., McTear M., Raman T., Stanney K., Su H., Wang Q., Lai J., Larson J., Oviatt S., Balaji T., Buisine S., Collings P., Cohen P., and Kraal B. Guidelines for multimodal user interface design. Communications of the ACM 47 1 (2004) 57-69
144. Rehg J., Loughlin M., and Waters K. Vision for a Smart Kiosk. Conference on Computer Vision and Pattern Recognition (CVPR) (1997) 90-696
145. Rosales R., and Sclaroff S. Learning body pose via specialized maps. NIPS 14 (2001) 1263-1270
146. Rosenblum M., Yacoob Y., and Davis L. Human expression recognition from motion using radial basis function network architecture. IEEE Transactions on Neurel Networks 7 5 (1996) 1121-1138
147. Ross A., and Jain A.K. Information Fusion in Biometrics. Pattern Recognition Letters, Special Issue on Multimodal Biometrics 24 13 (2003) 2115-2125
148. Roth P., and Pun T. Design and evaluation of a multimodal system for the non-visual exploration of digital pictures. INTERACT (2003)
149. Ruddaraju R., Haro A., Nagel K., Tran Q., Essa I., Abowd G., and Mynatt E. Perceptual user interfaces using vision-based eye tracking. ICMI (2003)
150. Salen K., and Zimmerman E. Rules of Play: Game Design Fundamentals (2003), MIT Press, Cambridge, MA
151. Santella A., and DeCarlo D. Robust clustering of eye movement recordings for quantification of visual interest. Eye Tracking Research and Applications (ETRA) (2004) 27-34
152. Sebe N., Cohen I., and Huang T.S. Multimodal emotion recognition. Handbook of Pattern Recognition and Computer Vision, World Scientific (2005)
153. Sebe N., Cohen I., Garg A., and Huang T.S. Machine Learning in Computer Vision (2005), Springer, Berlin, NY
154. Sebe N., Cohen I., Gevers T., and Huang T.S. Emotion Recognition Based on Joint Visual and Audio Cues. International Conference on Pattern Recognition (2006) 1136-1139
155. E. Schapira, R. Sharma, Experimental evaluation of vision and speech based multimodal interfaces, Workshop on Perceptive User Interfaces, pp. 1-9, 2001.
156. Schuller B., Lang M., and Rigoll G. Multimodal emotion recognition in audiovisual communication. ICME (2002)
157. Selker T. Visual attentive interfaces. BT Technology Journal 22 4 (2004) 146-150
158. A. Shaikh, S. Juth, A. Medl, I. Marsic, C. Kulikoswki, J. Flanagan, An architecture for multimodal information fusion, Workshop on Perceptual User Interfaces, 1997.
159. Sharma R., Yeasin M., Krahnstoever N., Rauschert I., Cai G., Brewer I., MacEachren A., and Sengupta K. Speech-gesture driven multimodal interfaces for crisis management. Proceedings of the IEEE 91 9 (2003) 1327-1354
160. Shneiderman B. Direct manipulation for comprehensible, predictable, and controllable user interface. International Conference on Intelligent User Interfaces (1997)
161. Shneiderman B. Leonardo's Laptop: Human Needs and the New Computing Technologies (2002), MIT Press, Cambridge, MA
162. Sibert L.E., and Jacob R.J.K. Evaluation of eye gaze interaction. ACM Conference Human Factors in Computing Systems (CHI) (2000) 281-288
163. de Silva L.C., and Ng P. Bimodal emotion recognition. International Conference on Face and Gesture Recognition (2000)
164. Simpson R., LoPresti E., Hayashi S., Nourbakhsh I., and Miller D. The smart wheelchair component system. Journal of Rehabilitation Research and Development (2004)
165. Smith P., Shah M., and Lobo N.d.V. Determining driver visual attention with one camera. IEEE Transactions on Intelligent Transportation Systems 4 4 (2003)
166. Song M., Bu J., Chen C., and Li N. Audio-visual based emotion recognition: a new approach. CVPR (2004)
167. Sparacino F. The museum wearable: real-time sensor-driven understanding of visitors' interests for personalized visually-augmented museum experiences. Museums and the Web (2002)
168. In: Stock O., and Zancanaro M. (Eds). Multimodal Intelligent Information Presentation. Series Text. Speech and Language Technology vol. 27 (2005), Kluwer Academic, Dordrecht 325-340
169. Trivedi M.M., Cheng S.Y., Childers E.M.C., and Krotosky S.J. Occupant posture analysis with stereo and thermal infrared video: algorithms and experimental evaluation. IEEE Transactions on Vehicular Technology 53 6 (2004) 1698-1712
170. Trumbly J., Arnett K., and Johnson P. Productivity gains via an adaptive user interface. Journal of Human-computer Studies 40 (1994) 63-81
171. M. Turk, Gesture recognition, Handbook of Virtual Environment Technology, K. Stanney (Ed.), 2001.
172. Turk M. Computer vision in the interface. Communications of the ACM 47 1 (2004) 60-67
173. Turk M., and Robertson G. Perceptual Interfaces. Communications of the ACM 43 3 (2000) 32-34
174. Turk M., and Kölsch M. Perceptual interfaces. In: Medioni G., and Kang S.B. (Eds). Emerging Topics in Computer Vision (2004), Prentice Hall, Englewood Cliffs, NJ
175. Turk M. Multimodal human-computer interaction. In: Kisancanin B., Pavlovic V., and Huang T.S. (Eds). Real-time Vision for Human-computer Interaction (2005), Springer, Berlin, NY
176. Turunen M., and Hakulinen J. Jaspis2-an architecture for supporting distributed spoken dialogs. Eurospeech (2003)
177. R. Vertegaal (Ed.), Attentive user interfaces: Special issue, Communications of the ACM 46(3) (2003).
178. Wang J.-G., Sung E., and Venkateswarlu R. Eye gaze estimation from a single image of one eye. ICCV (2003) 136-143
179. Wang J.J.L., and Singh S. Video analysis of human dynamics-a survey. Real-Time Imaging 9 5 (2003) 321-346
180. Wang L., Hu W., and Tan T. Recent developments in human motion analysis. Pattern Recognition 36 (2003) 585-601
181. Wassermann K.C., Eng K., Verschure P.F.M.J., and Manzolli J. Live soundscape composition based on synthetic emotions. IEEE Multimedia Magazine 10 4 (2003)
182. Weiser M. Some computer science issues in ubiquitous computing. Communications of the ACM 36 7 (1993) 74-83
183. Y. Wu, T.S. Huang, Vision-based gesture recognition: a review, 3rd Gesture Workshop, 1999.
184. Wu Y., and Huang T.S. Human hand modeling, analysis and animation in the context of human computer interaction. IEEE Signal Processing 18 3 (2001) 51-60
185. Wu Y., Hua G., and Yu T. Tracking articulated body by dynamic Markov network. ICCV (2003) 1094-1101
186. Yacoob Y., and Davis L. Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 18 6 (1996) 636-642
187. Yang M.-H., Kriegman D., and Ahuja N. Detecting faces in images: a survey. IEEE Transactions on PAMI 24 1 (2002) 34-58
188. Yoshikawa H. Modeling humans in human-computer interaction. Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications (2002) 118-146
189. Y. Yoshitomi, S. Kim, T. Kawano, T. Kitazoe, Effect of sensor fusion for recognition of emotional states using voice, face image, and thermal image of face, International Workshop on Robot-Human Interaction, 2000.
190. Yu C., and Ballard D.H. A multimodal learning interface for grounding spoken language in sensorimotor experience. ACM Transactions on Applied Perception (2004)
191. Yuan Q., Sclaroff S., and Athitsos V. Automatic 2D hand tracking in video sequences. IEEE Workshop on Applications of Computer Vision (2005)
192. Zeng Z., Tu J., Liu M., Zhang T., Rizzolo N., Zhang Z., Huang T.S., Roth D., and Levinson S. Bimodal HCI-related affect recognition. ICMI (2004)
193. Zhao W., Chellappa R., Rosenfeld A., and Phillips J. Face recognition: a literature survey. ACM Computing Surveys 12 (2003) 399-458