Multimodal human computer interaction: A survey

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 3766 LNCS, Issue , 2005, Pages 1-15

  Jaímes, Alej́andro ^a   Sebe, Nícu ^b  

^a XEROX CORPORATION   (United States)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; GESTURE RECOGNITION; INTERACTIVE COMPUTER SYSTEMS; MODAL ANALYSIS; PATTERN RECOGNITION SYSTEMS; SENSOR DATA FUSION;

EMOTION; FACIAL EXPRESSION RECOGNITION; TASK MODELING;

HUMAN COMPUTER INTERACTION;

EID: 33646711871     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/11573425_1     Document Type: Conference Paper

References (95)

1. J.K. Aggarwal and Q. Cai, "Human motion analysis: A review," CVIU, 73(3):428-440, 1999.
2. Application of Affective Computing in Human-computer Interaction, Int. J. of Human-Computer Studies, 59(1-2), 2003.
3. J. Ben-Arie, Z. Wang, P. Pandit, and S. Rajaram, "Human activity recognition using multi-dimensional indexing," IEEE Trans. On PAMI, 24(8): 1091-1104, 2002.
4. M. Benali-Khoudja, M. Hafez, J.-M. Alexandre, and A. Kheddar, "Tactile interfaces: a state-of-the-art survey," Int. Symposium on Robotics, 2004.
5. A.F. Bobick and J. Davis, "The recognition of human movement using temporal templates," IEEE Trans. on PAMI, 23(3):257-267, 2001.
6. S.A. Brewster, J. Lumsden, M. Bell, M. Hall, S. and Tasker," Multimodal 'Eyes-Free' Interaction Techniques for Wearable Devices," in proc. ACM CHI 2003.
7. C. S. Campbell and P.P. Maglio, "A Robust Algorithm for Reading Detection," ACM Workshop on Perceptive User Interfaces, 2001.
8. P.R. Cohen, D.R. McGee, "Tangible Multimodal Interfaces for Safety-critical Applications," Communications of the ACM, Vol. 47, Issue 1, pp. 41-46, January 2004.
9. I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T.S. Huang, "Semi-supervised learning of classifiers: Theory, algorithms, and their applications to human-computer interaction," IEEE Trans. on PAMI, 22(12):1553-1567, 2004.
10. I. Cohen, N. Sebe, A. Garg, L. Chen, and T.S. Huang, "Facial expression recognition from video sequences: Temporal and static modeling," CVIU, 91(1-2): 160-187, 2003.
11. L.S. Chen, Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction, PhD thesis, UTUC, 2000.
12. A.T. Duchowski, "A Breadth-First Survey of Eye Tracking Applications," Behavior Research Methods, Instruments, and Computing, 34(4):455-70, 2002.
13. C. Dickie, R. Vertegaal, D. Fono, C. Sohn, D. Chen, D. Cheng, J.S. Shell and O. Aoudeh, "Augmenting and Sharing Memory with eyeBlog," in CARPE 2004.
14. Z. Duric, W. Gray, R. Heishman, F. Li, A. Rosenfeld, M. Schoelles, C. Schunn, and H. Wechsler, "Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction," Proc. of the IEEE, 90(7): 1272-1289, 2002.
15. P. Ekman, ed., Emotion in the Human Face, Cambridge University Press, 1982.
16. C. Fagiani, M. Betke, and J. Gips, "Evaluation of tracking methods for human-computer interaction," IEEE Workshop on Applications in Computer Vision, 2002.
17. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey" Patt. Recogn., 36:259-275, 2003.
18. T. Fong, I. Nourbakhsh, K. Dautenhahn, "A survey of socially interactive robots," Robotics and Autonomous Systems, 42(3-4): 143-166, 2003.
19. S. Fussell, L. Setlock, J. Yang, J. Ou, E. Mauer, A. Kramer, "Gestures over video streams to support remote collaboration on physical tasks," Human-Computer Int., 19(3):273-309, 2004.
20. A. Garg, M. Naphade, and T.S. Huang, "Modeling video using input/output Markov models with application to multi-modal event detection," Handbook of Video Databases: Design and Applications, 2003.
21. I. Garg, V. Pavlovic, and J. Rehg. "Boosted learning in dynamic Bayesian networks for multimodal speaker detection," Proceedings of the IEEE, 91 (9): 1355-1369, 2003.
22. D.M. Gavrila, "The Visual Analysis of Human Movement: A Survey," CVIU, 73(1): 82-98, 1999.
23. A. Hanjalic and L-Q. Xu, "Affective video content representation and modeling," IEEE Trans. on Multimedia, 7(1): 143-154, 2005.
24. A. Hakeem and M. Shah, "Ontology and taxonomy collaborated framework for meeting classification," ICPR, 2004.
25. R. Heishman, Z. Duric, and H. Wechsler, "Using eye region biometrics to reveal affective and cognitive states," CVPR Workshop on Face Processing in Video, 2004.
26. E. Hjelmas and B. K. Low, "Face detection: A survey," CVIU, 83:236-274, 2001.
27. W. Hu, T. Tan, L. Wang, and S. Maybank, "A Survey on Visual Surveillance of Object Motion and Behaviors", in IEEE Trans. On Systems, Man, and Cybernetics, Vol. 34, No. 3, Aug. 2004.
28. S. Intille, K. Larson, J. Beaudin, J. Nawyn, E. Tapia, P. Kaushik, "A living laboratory for the design and evaluation of ubiquitous computing technologies," Conf. on Human Factors in Computing Systems, 2004.
29. A. Jaimes and J. Liu, "Hotspot Components for Gesture-Based Interaction," in proc. IFIP Interact 2005, Rome, Italy, Sept. 2005.
30. R. El Kaliouby and P. Robinson: Real time inference of complex mental states from facial expressions and head gestures, CVPR Workshop on Real-time Vision for HCI, 2004.
31. S. Kettebekov and R. Sharma, "Understanding gestures in multimodal human computer interaction," Int. J. on Artificial Intelligence Tools, 9(2):205-223, 2000.
32. T. Kirishima, K. Sato, and K. Chihara, "Real-time gesture recognition by learning and selective control of visual interest points," IEEE Trans. on PAMI, 27(3):351-364, 2005.
33. T. Kisacanin, V. Pavlovic, and T.S.Huang (eds.). Real-Time Vision for Human-Computer Interaction, Springer-Verlag, New York, 2005.
34. Y. Kuno, N. Shimada, and Y. Shirai, "Look where you're going: A robotic wheelchair based on the integration of human and environmental observations," IEEE Robotics and Automation, 10(1):26-34, 2003.
35. P. Lang, "The emotion probe: Studies of motivation and attention," American Psychologist, 50(5):372-385, 1995.
36. A. Legin, A. Rudnitskaya, B. Seleznev, Yu. Vlasov, "Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie," Analytica Chimica Acta, Vol. 534, pp. 129-135, April 2005.
37. M.J. Lyons, M. Haehnel, and N. Tetsutani, "Designing, playing, and performing, with a vision-based mouth Interface," Conf. on New Interfaces for Musical Expression, 2003.
38. S. Marcel, "Gestures for multi-modal interfaces: A Review," Technical Report IDIAP-RR 02-34, 2002.
39. D. Maynes-Aminzade, R. Pausch, and S. Seitz, "Techniques for interactive audience participation," ICMI 2002.
40. I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, and D. Zhang, "Automatic analysis of multimodal group actions in meetings," IEEE Trans. on PAMI, 27(3):305-317, 2005.
41. D. McNeill, Hand and Mind: What Gestures Reveal About Thought, Univ. of Chicago Press, Chicago, IL, 1992.
42. A. Mehrabian, "Communication without words," Psychology Today, 2(4):53-56, 1968.
43. S. Meyer and A. Rakotonirainy, "A Survey of research on context-aware homes," Austral-asian Information Security Workshop Conference on ACSW Frontiers, 2003
44. T. B. Moeslund and E. Granum, "A survey of computer vision-based human motion capture," CVIU, 81(3):231-258, 2001.
45. J. Nielsen, "Non-command user interfaces," Comm. of the ACM, 36(4):83-99, 1993.
46. P. Y. Oudeyer, "The production and recognition of emotions in speech: Features and algorithms," Int. J. of Human-Computer Studies, 59(1-2): 157-183, 2003.
47. A. Oulasvirta and A. Salovaara, "A cognitive meta-analysis of design approaches to interruptions in intelligent environments," in Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'04): Extended Abstracts (2004).
48. P. Qvarfordt, and S. Zhai, "Conversing with the user based on eye-gaze patterns," Conf. Human-Factors in Computing Syst., 2005.
49. S. Oviatt, T. Darrell, and M. Flickner, "Multimodal Interfaces that Flex, Adapt, and Persist," (eds.) special issue, Communications of the ACM, Vol. 47, Issue 1, January 2004.
50. S.L. Oviatt and P. Cohen, "Multimodal interfaces that process what comes naturally," Comm. of the ACM, 43(3):45-48, 2000.
51. S.L. Oviatt, "Multimodal interfaces," Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, chap. 14, 286-304, 2003.
52. S.L. Oviatt, P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro, "Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions," Human-Computer Int., 15:263-322, 2000.
53. H. Pan, Z.P. Liang, T.J. Anastasio, and T.S. Huang. "Exploiting the dependencies in information fusion," CVPR, vol. 2:407-412, 1999.
54. M. Pantic and L.J.M. Rothkrantz, "Automatic analysis of facial expressions: The state of the art," IEEE Trans. on PAMI, 22(12): 1424-1445, 2000.
55. M. Pantic and L.J.M. Rothkrantz, "Toward an affect-sensitive multimodal human-computer interaction," Proceedings of the IEEE, 91(9):1370-1390, 2003.
56. J. Paradiso and F. Sparacino, "Optical Tracking for Music and Dance Performance," Optical 3-D Measurement Techniques IV, A. Gruen, H. Kahmen, eds., pp. 11-18, 1997.
57. V.I. Pavlovic, R. Sharma and T.S. Huang, "Visual interpretation of hand gestures for human-computer interaction: a review", IEEE Trans. on PAMI, 19(7):677-695, 1997.
58. J.B. Pelz, "Portable eye-tracking in natural behavior," J. of Vision, 4(11), 2004.
59. A. Pentland, "Looking at People," Comm. of the ACM," 43(3):35-44, 2000.
60. A. Pentland, "Socially Aware Computation and Communication," IEEE Computer Vol 38,No. 3, March 2005.
61. R. W. Picard, Affective Computing, MIT Press, 1997.
62. M. Porta, "Vision-based user interfaces: methods and applications," Int. J. Human-Computer Studies, 57(1):27-73, 2002.
63. L. M. Reeves et. al., "Guidelines for multimodal user interface design," Communications of the ACM, Vol. 47, Issue 1, pp. 57-69, January 2004.
64. R. Resales and S. Sclaroff, "Learning body pose via specialized maps," NIPS, Vol. 14, pp 1263-1270, 2001.
65. P. Roth, and T. Pun, "Design and evaluation of a multimodal system for the non-visual exploration of digital pictures," INTERACT 2003.
66. R. Ruddaraju, A. Haro, K. Nagel, Q. Tran, I. Essa, G. Abowd, E. Mynatt, "Perceptual user interfaces using vision-based eye tracking," ICMI, 2003.
67. A. Santella and D. DeCarlo, "Robust clustering of eye movement recordings for quantification of visual interest," Eye Tracking Research and Applications (ETRA), pp. 27-34, 2004.
68. N. Sebe, I. Cohen, and T.S. Huang, Multimodal emotion recognition, Handbook of Pattern Recognition and Computer Vision, World Scientific, 2005.
69. E. Schapira and R. Sharma, "Experimental evaluation of vision and speech based multimodal interfaces," Workshop on Perceptive User Interfaces, pp. 1-9, 2001.
70. B. Schuller, M. Lang, G. Rigoll: "Multimodal emotion recognition in audiovisual communication," ICME, 2002.
71. T. Selker, "Visual Attentive Interfaces," BT Technology Journal, Vol. 22, No. 4, pp. 146-150, Oct. 2004.
72. R. Sharma, M. Yeasin, N. Krahnstoever, I. Rauschert, G. Cai, I. Brewer, A. MacEachren, and K. Sengupta, "Speech-gesture driven multimodal interfaces for crisis management," Proceedings of the IEEE, 91(9):1327-1354, 2003.
73. L.E. Sibert and R.J.K. Jacob, "Evaluation of eye gaze interaction," Conf. Human-Factors in Computing Syst., pp. 281-288, 2000.
74. P. Smith, M. Shah, and N.d.V. Lobo, "Determining driver visual zttention with one camera," IEEE Trans. on Intelligent Transportation Systems, 4(4), 2003.
75. R. Simpson, E. LoPresti, S. Hayashi, I. Nourbakhsh, D. Miller, "The smart wheelchair component system," J. of Rehabilitation Research and Development, May/June 2004.
76. F. Sparacino, "The museum wearable: Real-time sensor-driven understanding of visitors' interests for personalized visually-augmented museum experiences," Museums and the Web, 2002.
77. M. M. Trivedi, S. Y. Cheng, E. M. C. Childers, S. J. Krotosky, "Occupant posture analysis with stereo and thermal infrared video: Algorithms and experimental evaluation," IEEE Trans. on Vehicular Technology, 53(6):1698-1712, 2004.
78. M. Turk, "Gesture recognition," Handbook of Virtual Environment Technology, K. Stanney (ed.), 2001.
79. M. Turk, "Computer vision in the interface," Communications of the ACM, Vol. 47, Issue 1, pp. 60-67, January 2004.
80. M. Turk and G. Robertson, "Perceptual Interfaces," Communications of the ACM, Vol. 43, Issue 3, pp. 32-34, 2000.
81. M. Turk and M. Kölsch, "Perceptual Interfaces," G. Medioni and S.B. Kang (eds.), Emerging Topics in Computer Vision, Prentice Hall, 2004.
82. J.-G. Wang, E. Sung, and R. Venkateswarlu, "Eye gaze estimation from a single image of one eye," ICCV, pp. 136-143, 2003.
83. L. Wang, W. Hu and T. Tan "Recent developments in human motion analysis," Patt. Recogn., 36 (2003) 585-601
84. J.J.L. Wang and S. Singh, "Video analysis of human dynamics - A survey," Real-Time Imaging, 9(5):321-346, 2003.
85. K.C. Wassermann, K. Eng, P.F.M.J. Verschure, and J. Manzolli, "Live soundscape composition based on synthetic emotions," IEEE Multimedia Magazine, 10(4), 2003.
86. rd Gesture Workshop, 1999.
87. Y. Wu, G. Hua and T. Yu, "Tracking articulated body by dynamic Markov network," ICCV, pp. 1094-1101, 2003.
88. M.-H. Yang, D. Kriegman, and N. Ahuja, "Detecting faces in images: A survey," IEEE Trans. on PAMI, 24(1):34-58, 2002.
89. Q. Yuan, S. Sclaroff and V. Athitsos, "Automatic 2D hand tracking in video sequences," IEEE Workshop on Applications of Computer Vision, 2005.
90. C. Yu and D. H. Ballard, "A multimodal learning interface for grounding spoken language in sensorimotor experience", ACM Trans. on Applied Perception, 2004.
91. W. Zhao, R. Chellappa, A. Rosenfeld, and J. Phillips, "Face recognition: A literature survey" ACM Computing Surveys, 12:399-458, 2003.
92. K. Salen and E. Zimmerman, Rules of Play: Game Design Fundamentals, MIT Press, 2003.
93. Z. Zeng, J. Tu, M. Liu, T. Zhang, N. Rizzolo, Z. Zhang, T.S. Huang, D. Roth, and S. Levinson, "Bimodal HCI-related affect recognition," ICMI, 2004.
94. Y. Wu and T.S. Huang, "Human hand modeling, analysis and animation in the context of human computer interaction," IEEE Signal Processing, 18(3):51-60, 2001.
95. I.R. Murray and J.L. Arnott, "Toward the simulation of emotion in synthetic speech: A review of the literature of human vocal emotion," J. of the Acoustic Society of America, 93(2): 1097-1108, 1993.