Toward Multimodal Human-Computer Interface

Proceedings of the IEEE

Volumn 86, Issue 5, 1998, Pages 853-869

  Sharma, Rajeev ^a,b   Pavlovic, Vladimir I ^a   Huang, Thomas S ^a  

^a The Pennsylvania State University   (United States)

^b University of Illinois at Urbana Champaign   (United States)

Author keywords

Human computer interface; Multimodality; Sensor fusion

Indexed keywords

COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; DECISION THEORY; IMAGE ANALYSIS; INTEGRATION; INTERACTIVE COMPUTER SYSTEMS; MODAL ANALYSIS; PATTERN RECOGNITION; SENSOR DATA FUSION; VIDEO SIGNAL PROCESSING; VIRTUAL REALITY;

MODALITY INTEGRATION; VISION BASED GESTURE RECOGNITION;

HUMAN COMPUTER INTERACTION;

EID: 0032075723     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/5.664275     Document Type: Article

References (94)

1. J. A. Adam, "Virtual reality," IEEE Spectrum, vol. 30, no. 10, pp. 22-29, 1993.
2. H. Rheingold, Virtual Reality. New York: Summit Books, 1991.
3. A; G. Hauptmann and P. McAvinney, "Gesture with speech for graphics manipulation," Int. J. Man-Machine Studies, vol. 38, pp. 231-249, Feb. 1993.
4. S. Mann, "Wearable computing: A first step toward personal imaging," IEEE Computer Mag., vol. 30, pp. 25-32, Feb. 1997.
5. R. W. Pickard and J. Healey, "Affective wearables," in Proc. Int. Symp. Wearable Computing, Cambridge, MA, Oct. 1997.
6. L. R. Rabiner and B. Juang, Fundamentals of Speech Recognition. Englewood Cliffs, NJ: Prentice-Hall, 1993.
7. S. Oviatt, A. DeAngeli, and K. Kühn, "Integration and synchronization of input modes during multimodal human-computer interaction," in Proc. Conf. Human Factors in Computing Systems (CHI'97), Atlanta, GA, pp. 415-422.
8. D. L. Hall and J. Llinas, "An introduction to multisensor data fusion," Proc. IEEE, vol. 85, pp. 6-23, Jan. 1997.
9. R. R. Murphy, "Biological and cognitive foundations of intelligent data fusion," IEEE Trans. Syst., Man, Cybern., vol. 26, pp. 42-51, Jan. 1996.
10. D. Salber and J. Coutaz, "Applying the Wizard of Oz technique to the study of multimodal systems," in Proc. EWHCI 93, Moscow, Russia, 1993.
11. P. R. Cohen, M. Darlymple, F. C. N. Pereira, J. W. Sullivan, R. A. Gargan, Jr., J. L. Schlossberg, and S. W. Tyler, "Synergic use of direct manipulation and natural language," in Proc. Conf. Human Factors in Computing Systems (CHI'89), Austin, TX, pp. 227-233.
12. V. I. Pavlovic, R. Sharma, and T. S. Huang, "Visual interpretation of hand gestures for human-computer interaction: A review," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, no. 7, pp. 677-695, 1997.
13. A. Waibel, M. T. Vo, P. Duchnowski, and S. Manke, "Multimodal interfaces," Artificial Intell. Rev., vol. 10, pp. 299-319, Aug. 1995.
14. B. A. Myers, "A brief history of human computer interaction technology," Human Computer Interaction Institute, School of Computer Science, Carnegie-Mellon University, Pittsburgh, PA, Tech. Rep. CMU-CS-TR-96-163, 1996.
15. F. H. Raab, E. B. Blood, T. O. Steiner, and H. R. Jones, "Magnetic position and orientation tracking system," IEEE Trans. Aerosp. Electron. Syst., vol. 15, pp. 709-718, 1979.
16. R. Azuma, 'Tracking requirements for augmented reality," Commun. ACM, vol. 36, no. 7, pp. 50-52, 1993.
17. T. Baudel and M. Baudouin-Lafon, "Charade: Remote control of objects using free-hand gestures," Commun. ACM, vol. 36, no. 7, pp. 28-35, 1993.
18. S. S. Fels and G. E. Hinton, "Glove-Talk: A neural network interface between a Data-Glove and a speech synthesizer," IEEE Trans. Neural Neftvorks, vol. 4, pp. 2-8, Jan. 1993.
19. D. J. Sturman and D. Zeltzer, "A survey of glove-based input," IEEE Comput. Graph. Applicat. Mag., vol. 14, pp. 30-39, Jan. 1994.
20. D. L. Quam, "Gesture recognition with a DataGIove," in Proc. 1990 IEEE National Aerospace and Electronics Conf., 1990, vol. 2.
21. C. Wang and D. J. Cannon, "A virtual end-effector pointing System in point-and-dircct robotics for inspection of surface flaws using a neural network based skeleton transform," in Proc. IEEE Int. Conf. Robotics and Automation, May 1993, vol. 3, pp. 784-789.
22. B. H. Juang, "Speech recognition in adverse environments," Comput. Speech Language, vol. 5, pp. 275-294, 1991.
23. D. Stork and H.-L. Lu, "Spcechreading by Boltzmann zippers," in Machines that Learn. Snowbird, UT: 1996.
24. R. Sharma, T. S. Huang, V. I. Pavlovic, Y. Zhao, Z. Lo, S. Chu, K. Schulten, A. Dalke, J. Phillips, M. Zeller, and W. Humphrey, . "Speech/gesture interface to a visual computing environment for molecular biologists," in Proc. Int. Conf. Pattern Recognition, Aug. 1996, pp. 964-968.
25. F. Hatfield, E. A. Jenkins, M. W. Jennings, and G. Calhoun, "Principles and guidelines for the design of eye/voice interaction dialogs," in Proc. 3rd Ann. Symp. Human Interaction with Complex Systems, Dayton, OH, 1996, pp. 10-19.
26. T. E. Hutchinson, "Computers that sense eye position on the display," Computer, vol. 26, pp. 65-67, July 1993.
27. R. J. K. Jacob, "What you look at is what you get," Computer, vol. 26, pp. 65-67, July 1993.
28. I. A. Essa and A. P. Pentland, "Coding analysis, interpretation, and recognition of facial expressions," IEEE Trans. Pattern Anal. Machine Intel!., vol. 19, no. 7, pp. 757-763, 1997.
29. D. M. Gavrila and L. S. Davis, "Toward 3-D model-based tracking and recognition of human movement: A multi-view approach," in Proc. 1WAFGR'95, Zurich, Switzerland, June 1995, pp. 272-277.
30. C. R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland, "Pfinder: Real-time tracking of the human body," IEEE Trans. Pattern Anal. Machine Inteli, vol. 19, no. 7, pp. 780-785, 1997.
31. C. Nodine, H. Kundel, L. Toto, and E. Krupinski, "Recording and analyzing eye-position data using a microcomputer workstation," Behavior Res. Methods, Instruments Comput., vol. 24, no. 3, pp. 475-584, 1992.
32. C. Lansing and G. W. McConkie, "A new method for speechreading research: Tracking observers' eye movements," J. Acad. Rehabilitative Attdiology, vol. 28, pp. 25-43, 1994.
33. V. I. Pavlovic, R. Sharma, and T. S. Huang, "Gestural interface to a visual computing environment for molecular biologists," in Proc. Int. Conf. Automatic Face and Gesture Recognition, Killington, VT, Oct. 1996, pp. 30-35.
34. R. Sharma and J. Molineros, "Computer vision-based augmented reality for guiding manual assembly," Presence: Teleoperators Virtual Environ., vol. 6, pp. 292-317, June 1997.
35. F. K. H. Quek, "Eyes in the interface," Image Vision Comput., vol. 13, Aug. 1995.
36. R. Sharma, T. S. Huang, and V. I. Pavlovic, "A multimodal framework for interacting with virtual environments," in Human Interaction with Complex Systems, C. A. Ntuen, E. H. Park, and J. H. Kirn, Eds. Norwell, MA: Kluwer, 1996, pp. 53-71.
37. R. M. Satava and S. B. Jones, "Virtual environments for medical training and education," Presence: Teleoperators Virtual Environ., vol. 6, no. 2, pp. 139-146, 1997.
38. S. L. Delp, P. Loan, C. Basdogan, and J. M. Rosen, "Surgical simulation: An emerging technology for training in emergency medicine," Presence: Teleoperators Virtual Environ., vol. 6, no. 2, pp. 147-159, 1997.
39. M. Bergamasco, "Haptic interfaces: The study of force and tactile feedback systems," in Proc. IEEE Int. Workshop Robot on Robot and Human Communication, 1995, pp. 15-20.
40. T. R. Sheridan, Telerobotics, Automation, and Human Supervisory Control. Cambridge, MA: MIT Press, 1992.
41. Z. A. Keirn and J. I. Aunon, "Man-machine communications through brain-wave processing," IEEE Eng. Med. Biology Mag., vol. 9, pp. 55-57, 1990.
42. D. J. McFarland, G. W. Neat, R. F. Read, and J. R. Wolpaw, "An EEG-based method for graded cursor control," Psychobiology, vol. 21, pp. 77-81, 1993.
43. V. T. Nasman, G. L. Calhoun, and G. R. McMillan, "Brainactuated control and HMDS," in Head Mounted Displays, New York: McGraw-Hill, 1997, pp. 285-312.
44. W. Putnam and R. B. Knapp, "Real-time computer control using pattern recognition of the electromyogram," in Proceedings of the Fifteenth Annual International Conference on Engineering in Medicine and Biology Society. New York: IEEE Press, 1993, vol. 15, pp. 1236-1237.
45. H. S. Lusted and R. B. Knapp, "Controlling computers with neural signals," Sei. Amer., pp. 82-87, Oct. 1996.
46. T. Elbert, B. Rockstroh, W. Lutzenbcrger, and W. Birbaumer, Self-Regulation of the Brain and Behavior. New York: Springer-Verlag, 1984.
47. S. Suryanarayanan and N. R. Reddy, "EMG-based interface for position tracking and control in VR environments and teleoperation," Presence: Teleoperators Virtual Environ., vol. 6, no. 3, pp. 282-291, 1997.
48. A. M. Junker, J. H. Schnurer, D. F. Ingle, and C. W. Downey, "Loop-closure of the visual cortex response," Armstrong Aerospace Medical Research Laboratory, Wright-Patterson Air Force Base, Dayton, OH, Tech. Rep. AAMRL-TR-88-014, 1988.
49. D. W. Patmore and R. B. Knapp, "A cursor controller using evoked potentials and EOG," in Proc. RESNA 18th Ann. Conf., 1995, pp. 702-704.
50. D. McNeill, Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
51. J. Streeck, "Gesture as communication I: Its coordination with gaze and speech," Commun. Monographs, vol. 60, pp. 275-299, Dec. 1993.
52. B. V. Dasarathy, "Sensor fusion potential exploitation-Innovative architectures and illustrative approaches," Proc. IEEE, vol. 85, pp. 24-38, Jan. 1997.
53. B. Stein and M. A. Meredith, The Merging of Senses. Cambridge, MA: MIT Press, 1993.
54. T. G. R. Bower, 'The evolution of sensory system," in Perception: Essays in Honor of James J. Gibson, R. B. MacLeod and H. L. Pick, Jr., Eds. Ithaca, NY: Cornell Univ. Press, 1974, pp. 141-153.
55. D. Heckerman, "A tutorial on learning with Bayesian networks," Microsoft Corp., Seattle, WA, Tech. Rep. MSR-TR95-06, Mar. 1995.
56. P. K. Varshney, Distributed Detection and Data Fusion. New York: Springer-Verlag, 1996.
57. R. G. Brown and P. Y. C. Hwang, Introduction to Random Signals and Kaiman Filtering. New York: Wiley, 1992.
58. C. K. Chui and G. Chen, Kaiman Filtering with Real-Time Applications. Berlin/Heidelberg, Germany: Springer-Verlag, 1991.
59. K. C. Chou, A. S. Willsky, and A. Benveniste, "Multiscale recursive estimation, data fusion, and regularization," IEEE Trans. Automat. Contr., vol. 39, pp. 464-478, Mar. 1994.
60. M. I. Jordan and C. M. Bishop, "Neural networks," in CRC Handbook of Computer Science A. Tucker, Ed. Boca Raton, FL: CRC Press, 1996.
61. H. A. Boulard and N. Morgan, Connectionist Speech Recognition. A Hybrid Approach. Norwell, MA: Kluwer, 1994.
62. A. K. Jain, J. Mao, and K. M. Mohiuddin, "Artificial neural networks: A tutorial," IEEE Computer Mag., pp. 31-44, Mar. 1996.
63. P. J. Werbos, "Generalization of back propagation with applications to a recurrent gas market model," Neural Nehvorks, vol. 1, pp. 339-356, 1988.
64. P. Haffner, M. Franzini, and A. Weibel, "Integrating time alignment and neural networks for high performance continuous speech recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Toronto, Ont., Canada, Apr. 1991, pp. 105-108.
65. U. Meier, W. Hurst, and P. Duchnowski, "Adaptive bimodal sensor fusion for automatic speechreading," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, 1996.
66. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE, vol. 77, pp. 257-286, Feb. 1989.
67. T. E. Starner and A. Pentland, "Visual recognition of American sign language using hidden Markov models," in Proc. Int. Workshop Automatic Face and Gesture Recognition, Zurich, Switzerland, June 1995, pp. 189-194.
68. L. W. Campbell, D. A. Becker, A. Azarbayejani, A. F. Bobick, and A. Pentland, "Invariant features for 3-D gesture recognition," in Proc. Int. Conf. Automatic Face and Gesture Recognition, Killington, VT, Oct. 1996, pp. 157-162.
69. D. X. Sun and L. Deng, "Non-stationary hidden Markov models for speech recognition," in Image Models (and Their Speech Model Cousins), S. E. Levinson and L. Shepp, Eds. New York: Springer-Verlag, 1996, pp. 161-182.
70. A. Adjoudani and C. Benoît, "Audio-visual speech recognition compared across two architectures," in Proc. Eurospeech'95 Conf., Madrid, Spain, 1995, vol. 2, pp. 1563-1566.
71. V. Pavlovie, G. Berry, and T. S. Huang, "Integration of audio/visual information for use in human-computer intelligent interaction," in Proc. IEEE Int. Conf. Image Processing, Santa Barbara, CA, 1997.
72. M. E. Hennecke, D. G. Stork, and K. V. Prasad, "Visionary speech: Looking ahead to practical speechreading systems," in Proc. Speechreading by Man and Machine: Models, Systems and Applications Workshop, AugVSept. 1995.
73. M. Minsky, "A framework for representing knowledge," in The Psychology of Computer Vision, P. H. Winston, Ed. New York: McGraw-Hill, 1975.
74. F. Lehman, Ed., Semantic Networks in Artificial Intelligence. Oxford, England: Pergamon, 1992.
75. J. Guan and D. A. Bell, Evidence Theory and its Applications, vol. 1. Amsterdam, The Netherlands: North-Holland, 1991.
76. R. A. Bolt, "Put that there: Voice and gesture at the graphics interface," ACM Comput. Graph., vol. 14, no. 3, pp. 262-270, 1980.
77. M. T. Vo and C. Wood, "Building an application framework for speech and pen input integration in multimodal learning interfaces," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, 1996, pp. 3545-3548.
78. R. Kober, U. Harz, and J. Schiffers, "Fusion of visual and acoustic signals for command-word recognition," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing, 1997.
79. J. M. Bradshaw, "An introduction to software agents," in Sofhvare Agents, J. M. Bradshaw, Ed. Cambridge, MA: AAAI Press/MIT Press, 1997.
80. Y. Shoham, "An overview of agent-oriented programming," in Sofavare Agents, J. M. Bradshaw, Ed. AAAI Press/MIT Press, 1997.
81. N. Negroponte, "Agents: From direct manipulation to delegation," in Sofhvare Agents, J. M. Bradshaw, Ed. AAAI Press/MIT Press, 1997.
82. D. B. Moran, A. J. Cheyer, L. E. Julia, D. L. Martin, and S. Park, "Multimodal user interface in the open agent architecture," in Proc. ACM Int. Conf. Intelligent User Interfaces, Orlando, FL, 1997, pp. 61-68.
83. J. A. Pittman, I. Smith, P. Cohen, S. Oviatt, and T.-C. Yang, "QuickSet: A multimodal interface for military simulation," in Proc. 6th Conf. Computer-Generated Forces and Behavioral Representation, Orlando, FL, 1996, pp. 217-224.
84. M. Nelson, W. Humphrey, A. Gursoy, A. Dalke, L. Kale, R. Skeel, K. Schulten, and R. Kufrin, "MDScope-A visual computing environment for structural biology," Comput. Phys. Commun., vol. 91, no. 1/2/3, pp. 111-134, 1995.
85. P. R. Cohen, M. Johnston, D. McGee, S. Oviatt, and J. Pittman, "QuickSet: Multimodal interaction for simulation set-up and control," in Proc. 5th Applied Natural Language Processing Meeting, Washington, DC, 1997.
86. B. Suhm, P. Geunter, T. Kemp, A. Lavie, L. Mayfield, A. McNair, I. Rogina, T. Schultz, T. Sloboda, W. Ward, M. Woszczyna, and A. Waibel, "Janus: Toward multilingual spoken language translation," in Proc. ARPA SLT Workshop, Austin, TX, 1995.
87. J. Wang, "Integration of eye-gaze, voice and manual response in multimodal user interface," in Proc. IEEE Int. Conf. Systems, Man, and Cybernetics, 1995, pp. 3938-3942.
88. M. Fukumoto, Y. Suenaga, and K. Mase, "'Finger-pointer': Pointing interface by image processing," Comput. Graph., vol. 18, no. 5, pp. 633-642, 1994.
89. C. Codclla, R. Jalili, L. Koved, et al., "Interactive simulation in a multi-person virtual world," in ACM Conf. Human Factors in Computing Systems (CHI'92), pp. 329-334.
90. P. Macs, T. Darrell, B. Blumberg, and A. Pentland, "The ALIVE system: Wireless, full-body interaction with autonomous agents," ACM Multimedia S\st., 1996.
91. A. Pentland, "Smart rooms," Sei. Amer., pp. 54-62, Apr. 1996.
92. MIT Media Laboratory, Perceptual Intelligence Group. [Online]. Available: http://casr.www.media.mit.edu/groups/casr/ pentland.html.
93. N. Tosa. (1996). Neuro-Baby. [Online]. Available: http://www.mic.atr.co.jp/tosa.
94. M. Kakimoto, N. Tosa, J. Mori, and A. Sanada, 'Tool of NeuroBaby," Inst. Television Eng. Jpn. Tech. Rep., vol. 16, pp. 7-12, June 1992.