Breaking the robustness barrier: Recent progress on the design of robust multimodal systems

Advances in Computers

Volumn 56, Issue C, 2002, Pages 305-341

  Oviatt, Sharon ^a  

^a OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77956782689     PISSN: 00652458     EISSN: None     Source Type: Book Series    
DOI: 10.1016/S0065-2458(02)80009-2     Document Type: Chapter

References (114)

1. Benoit C., Martin J.C., Pelachaud C., Schomaker L., and Suhm B. Audio-visual and multimodal speech-based systems. In: Gibbon D., Mertins I., and Moore R. (Eds). Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation (2000), Kluwer Academic, Boston 102-203
2. [Reprinted in. Oviatt S.L., Cohen P.R., Wu L., Vergo J., Duncan L., Suhm B., Bers J., Holzman T., Winograd T., Landay J., Larson J., and Ferro D. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research direction. Chap. 19. Human Computer Interaction 15 4 (2000) 263-322
3. [Reprinted in. Oviatt S.L., Cohen P.R., Wu L., Vergo J., Duncan L., Suhm B., Bers J., Holzman T., Winograd T., Landay J., Larson J., and Ferro D. Designing the user interface for multimodal speech and gesture applications: State-of-the-art systems and research direction. Chap. 19. In: Carroll J. (Ed). Human-Computer Interaction in the New Millennium (2001), Addison-Wesley, Reading, MA 421-456
4. Neti C., Iyengar G., Potamianos G., Senior A., and Maison B. Perceptual interfaces for information interaction: Joint processing of audio and visual information for human-computer interaction. Proceedings of the International Conference on Spoken Language Processing, Beijing 3 (2000) 11-14
5. In: Pankanti S., Bolle R.M., and Jain A. (Eds). Biometrics: The future of identification. Computer 33 (2000) 46-80 2
6. Benoit C., and Le Goff B. Audio-visual speech synthesis from French text: Eight years of models, designs and evaluation at the ICP. Speech Communication 26 (1998) 117-129
7. Cohen P.R., Johnston M., McGee D., Oviatt S., Pittman J., Smith I., Chen L., and Clow J. Quickset: Multimodal interaction for distributed applications. Proceedings of the Fifth ACM International Multimedia Conference (1997), ACM Press, New York 31-40
8. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines (1996), Springer-Verlag, New York
9. In: Turk M., and Robertson G. (Eds). Perceptual user interfaces. Communications of the ACM 43 3 (2000) 32-70
10. Zhai S., Morimoto C., and Ihde S. Manual and gaze input cascaded (MAGIC) pointing. Proceedings of the Conference on Human Factors in Computing Systems (CH'99) (1999), ACM Press, New York 246-253
11. Bolt R.A. Put-that-there: Voice and gesture at the graphics interface. Computer Graphics 14 3 (1980) 262-270
12. [Reprinted in Readings in Intelligent User Interfaces (Maybury and Wahlster, Eds.), pp. 29-37, Morgan Kaufmann, San Francisco.]. Cohen P.R., Dalrymple M., Moran D.B., Pereira F.C.N., Sullivan J.W., Gargan R.A., Schlossberg J.L., and Tyler S.W. Synergistic use of direct manipulation and natural language. Proceedings of the Conference on Human Factors in Computing Systems (CHI'89) (1989), ACM Press, New York 227-234
13. Kobsa A., Allgayer J., Reddig C., Reithinger N., Schmauks D., Harbusch K., and Wahlster W. Combining deictic gestures and natural language for referent identification. Proceedings of the 11th International Conf. on Computational Linguistics. Bonn, Germany (1986) 356-361
14. Neal J.G., and Shapiro S.C. Intelligent multimedia interface technology. In: Sullivan J.W., and Tyler S.W. (Eds). Intelligent User Interfaces (1991), ACM Press, New York 11-43
15. Seneff S., Goddeau D., Pao C., and Polifroni J. Multimodal discourse modeling in a multi-user multi-domain environment. In: Bunnell T., and Idsardi W. (Eds). Proceedings of the International Conference on Spoken Language Processing Vol. 1 (1996), University of Delaware and A. I. duPont Institute 192-195
16. Siroux J., Guyomard M., Multon F., and Remondeau C. Modeling and processing of the oral and tactile activities in the Georal tactile system. Proceedings of the International Conference on Cooperative Multimodal Communication, Theory & Applications. Eindhoven, Netherlands (1995)
17. Chap. 3. Wahlster W. User and discourse models for multimodal communication. In: Sullivan J.W., and Tyler S.W. (Eds). Intelligent User Interfaces (1991), ACM Press, New York 45-67
18. Oviatt S.L., and Cohen P.R. Multimodal systems that process what comes naturally. Communications of the ACM 43 3 (2000) 45-53
19. In: Rubin P., Vatikiotis-Bateson E., and Benoit C. (Eds). Speech Communication 26 (1998) 1-2
20. Oviatt S.L. Multimodal Interfaces. In: Jacko J., and Sears A. (Eds). Handbook of Human-Computer Interaction (2002), Lawrence Erlbaum, Mahwah, NJ
21. Oviatt S.L., Cohen P.R., Fong M.W., and Frank M.P. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. Proceedings of the International Conference on Spoken Language Processing, University of Alberta Vol. 2 (1992) 1351-1354
22. Bers J., Miller S., and Makhoul J. Designing conversational interfaces with multimodal interaction. DARPA Workshop on Broadcast News Understanding Systems (1998) 319-321
23. Cheyer A. MVIEWS: Multimodal tools for the video analyst. Proceedings of the International Conference on Intelligent User Interfaces (IUI'98) (1998), ACM Press, New York 55-62
24. Waibel A., Suhm B., Vo M.T., and Yang J. Multimodal interfaces for multimedia information agents. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (IEEE-ICASSP) Vol. 1 (1997), IEEE Press, Menlo Park, CA 167-170
25. Wu L., Oviatt S., and Cohen P. Multimodal integration: A statistical view. IEEE Transactions on Multimedia 1 4 (1999) 334-342
26. Bangalore S., and Johnston M. Integrating multimodal language processing with speech recognition. In: Yuan B., Huang T., and Tang X. (Eds). Proceedings of the International Conference on Spoken Language Processing (ICSLP'2000) Vol. 2 (2000), Chinese Friendship, Beijing 126-129
27. Denecke M., and Yang J. Partial information in multimodal dialogue. In: Yuan B., Huang T., and Tang X. (Eds). Proceedings of the International Conference on Spoken Language Processing (ICSLIP'2000) (2000), Chinese Friendship, Beijing 624-633
28. Bernstein L., and Benoit C. For speech perception by humans or machines, three senses are better than one. Proceedings of the International Conference on Spoken Language Processing 3 (1996) 1477-1480
29. Cohen M.M., and Massaro D.W. Modeling coarticulation in synthetic visible speech. In: Thalmann N.M., and Thalmann D. (Eds). Models and Techniques in Computer Animation (1993), Springer-Verlag, Berlin 139-156
30. Massaro D.W., and Stork D.G. Sensory integration and speechreading by humans and machines. American Scientist 86 (1998) 236-244
31. McGrath M., and Summerfield Q. Intermodal timing relations and audiovisual speech recognition by normal-hearing adults. Journal of the Acoustical Society of America 77 2 (1985) 678-685
32. McGurk H., and MacDonald J. Hearing lips and seeing voices. Nature 264 (1976) 746-748
33. McLeod A., and Summerfield Q. Quantifying the contribution of vision to speech perception in noise. British Journal of Audiology 21 (1987) 131-141
34. Robert-Ribes J., Schwartz J.L., Lallouache T., and Escudier P. Complementarity and synergy in bimodal speech: Auditory, visual, and audio-visual identification of French oral vowels in noise. Journal of the Acoustical Society of America 103 6 (1998) 3677-3689
35. Sumby W.H., and Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26 (1954) 212-215
36. Summerfield A.Q. Lipreading and audio-visual speech perception. Philosophical Transactions of the Royal Society of London, Series B 335 (1992) 71-78
37. Vatikiotis-Bateson E., Munhall K.G., Hirayama M., Lee Y.V., and Terzopoulos D. The dynamics of audiovisual behavior in speech. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 221-232
38. Petajan E.D. Automatic Lipreading to Enhance Speech Recognition. Ph.D. thesis (1984), University of Illinois at Urbana-Champaign
39. Brooke N.M., and Petajan E.D. Seeing speech: Investigations into the synthesis and recognition of visible speech movements using automatic image processing and computer graphics. Proceedings of the International Conference on Speech Input and Output: Techniques and Applications 258 (1986) 104-109
40. Adjoudani A., and Benoit C. Audio-visual speech recognition compared across two architectures. Proceedings of the Eurospeech Conference, Madrid, Spain Vol. 2 (1995) 1563-1566
41. Bregler C., and Konig Y. Eigenlips for robust speech recognition. Proceedings of the International Conference on Acoustics Speech and Signal Processing (IEEE-ICASSP) Vol. 2 (1994) 669-672
42. Goldschen A.J. Continuous Automatic Speech Recognition by Lipreading. Ph.D. thesis (1993), Department of Electrical Engineering and Computer Science, George Washington University
43. Silsbee P.L., and Su Q. Audiovisual sensory integration using Hidden Markov Models. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 489-504
44. Tomlinson M.J., Russell M.J., and Brooke N.M. Integrating audio and visual information to provide highly robust speech recognition. Proceedings of the International Conference on Acoustics Speech and Signal Processing (IEEE-ICASSP) (1996) 821-824
45. In: Cassell J., Sullivan J., Prevost S., and Churchill E. (Eds). Embodied conversational agents (2000), MIT Press, Cambridge, MA
46. Dupont S., and Luettin J. Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia 2 3 (2000) 141-151
47. Meier U., Hürst W., and Duchnowski P. Adaptive bimodal sensor fusion for automatic speechreading. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (IEEE-ICASSP) (1996), IEEE Press, Menlo Park, CA 833-836
48. Rogozan A., and Deglise P. Adaptive fusion of acoustic and visual sources for automatic speech recognition. Speech Communication 26 1-2 (1998) 149-161
49. Choudhury T., Clarkson B., Jebera T., and Pentland S. Multimodal person recognition using unconstrained audio and video. Proceedings of the 2nd International Conference on Audio-and-Video-based Biometric Person Authentication. Washington, DC (1999) 176-181
50. June 21. Lee J. Retooling products so all can use them. New York Times (2001)
51. Available. Jorge J., Heller R., and Guedj R. Proceedings of the NSF/EC Workshop on Universal Accessibility and Ubiquitous Computing: Providing for the Elderly. Alcácer do Sal, Portugal, 22-25 May (2001). http://immi.inesc.pt/alcacer01/procs/papers-list.html
52. Oviatt S.L., and van Gent R. Error resolution during multimodal humancomputer interaction. Proceedings of the International Conference on Spoken Language Processing Vol. 2 (1996), University of Delaware Press 204-207
53. (special issue on prosody and conversation). Oviatt S.L., Bernard J., and Levow G. Linguistic adaptation during error resolution with spoken and multimodal systems. Language and Speech 41 3-4 (1998) 419-442
54. Oviatt S.L. Mutual disambiguation of recognition errors in a multimodal architecture. Proceedings of the Conference on Human Factors in Computing Systems (CHI'99) (1999), ACM Press, New York 576-583
55. Rudnicky A., and Hauptman A. Multimodal interactions in speech systems. In: Blattner M., and Dannenberg R. (Eds). Multimedia Interface Design, Frontier Series (1992), ACM Press, New York 147-172
56. Suhm B. Multimodal Interactive Error Recovery for Non-conversational Speech User Interfaces. Ph.D. thesis (1998), Karlsruhe University, Germany
57. (special issue on multimodal interfaces). Oviatt S.L. Multimodal interactive maps: Designing for human performance. Human-Computer Interaction 12 (1997) 93-129
58. Oviatt S.L., and Kuhn K. Referential features and linguistic indirection in multimodal language. Proceedings of the International Conference on Spoken Language Processing, ASSTA Inc., Sydney, Australia Vol. 6 (1998) 2339-2342
59. Oviatt S.L. Multimodal system processing in mobile environments. Proceedings of the Thirteenth Annual ACM Symposium on User Interface Software Technology (UIST 2000) (2000), ACM Press, New York 21-30
60. (special issue on conversational interfaces). Oviatt S.L. Taming recognition errors with a multimodal architecture. Communications of the ACM 43 (2000) 45-51
61. Erber N.P. Auditory-visual perception of speech. Journal of Speech and Hearing Disorders 40 (1975) 481-492
62. Bregler C., Omohundro S.M., Shi J., and Konig Y. Towards a robust speechreading dialog system. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 409-423
63. Brooke M. Using the visual component in automatic speech recognition. Proceedings of the International Conference on Spoken Language Processing Vol. 3 (1996) 1656-1659
64. Nakamura S., Ito H., and Shikano K. Stream weight optimization of speech and lip, image sequence for audio-visual speech recognition. In: Yuan B., Huang T., and Tang X. (Eds). Proceedings of the International Conference on Spoken Language Processing (ICSLP 2000) Vol. 3 (2000), Chinese Friendship Publishers, Beijing 20-24
65. Potamianos G., and Neti C. Stream confidence estimation for audiovisual speech recognition. In: Yuan B., Huang T., and Tang X. (Eds). Proceedings of the International Conference on Spoken Language Processing (ICSLP 2000) Vol. 3 (2000), Chinese Friendship Publishers, Beijing 746-749
66. Silsbee P.L., and Bovik A.C. Computer lipreading for improved accuracy in automatic speech recognition. IEEE Transactions on Speech and Audio Processing 4 5 (1996) 337-351
67. Murphy R.R. Biological and cognitive foundations of intelligent sensor fusion. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans 26 1 (1996) 42-51
68. Lee D. The functions of vision. In: Pick H.L., and Saltzman E. (Eds). Modes of Perceiving and Processing information (1978), Wiley, New York 159-170
69. Pick H.L., and Saltzman E. Modes of perceiving and processing information. In: Pick Jr. H.L., and Saltzman E. (Eds). Modes of Perceiving and Processing Information (1978), Wiley, New York 1-20
70. Pick H. Information and effects of early perceptual experience. In: Eisenberg N. (Ed). Contemporary Topics in Developmental Psychology (1987), Wiley, New York 59-76
71. Stein B., and Meredith M. The Merging of the Senses (1993), MIT Press, Cambridge, MA
72. Welch R.B. Perceptual Modification: Adapting to Altered Sensory Environments (1978), Academic Press, New York
73. Bower T.G.R. The evolution of sensory systems. In: MacLeod R.B., and Pick Jr. H.L. (Eds). Perception: Essays in Honor of James J. Gibson (1974), Cornell University Press, Ithaca, NY 141-153
74. Freedman S.J., and Rekosh J.H. The functional integrity of spatial behavior. In: Freedman S.J. (Ed). The Neuropsychology of Spatially-Oriented Behavior (1968), Dorsey Press, Homewood, IL 153-162
75. Lackner J.R. Some aspects of sensory-motor control and adaptation in man. In: Walk R.D., and Pick H.L. (Eds). Intersensory Perception and Sensory Integration (1981), Plenum, New York 143-173
76. Hall D.L. Mathematical Techniques in Multisensor Data Fusion (1992), Artech House, Boston
77. Pavel M., and Sharma R.K. Model-based sensor fusion for aviation. Proceedings of SPIE 3088 (1997) 169-176
78. Hager G.D. Task-Directed Sensor Fusion and Planning: A Computational Approach (1990), Kluwer Academic, Boston
79. Martin A., Fiscus J., Fisher B., Pallet D., and Przybocki M. System descriptions and performance summary. Proceedings of the Conversational Speech Recognition Workshop/DARPA Hub-5E Evaluation (1997), Morgan Kaufman, San Mateo, CA
80. Weintraub M., Taussig K., Hunicke K., and Snodgrass A. Effect of speaking style on LVCSR performance. Proceedings of the Conversational Speech Recognition Workshop/DARPA Hub-5E Evaluation (1997), Morgan Kaufman, San Mateo, CA
81. Oviatt S.L., MacEachern M., and Levow G. Predicting hyperarticulate speech during human-computer error resolution. Speech Communication 24 (1998) 87-110
82. Banse R., and Scherer K. Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology 70 3 (1996) 614-636
83. Aist G., Chan P., Huang X., Jiang L., Kennedy R., Latimer D., Mostow J., and Yeung C. How effective is unsupervised data collection for children's speech recognition?. Proceedings of the International Conference on Spoken Language Processing, ASSTA Inc., Sydney Vol. 7 (1998) 3171-3174
84. Das S., Nix D., and Picheny M. Improvements in children's speech recognition performance. Proceedings of the International Conference on Acoustics, Speech and Signal Processing Vol. 1 (1998), IEEE Press, Menlo Park, CA 433-436
85. Potamianos A., Narayanan S., and Lee S. Automatic speech recognition for children. European Conference on Speech Communication and Technology 5 (1997) 2371-2374
86. Wilpon J.G., and Jacobsen C.N. A study of speech recognition for children and the elderly. Proceedings of the International Conference on Acoustic, Speech and Signal Processing (ICASSP'96) (1996) 349-352
87. Lee S., Potamianos A., and Narayanan S. Analysis of children's speech: Duration, pitch and formants. European Conference on Speech Communication and Technology Vol. 1 (1997) 473-476
88. In: Yeni-Komshian G., Kavanaugh J., and Ferguson C. (Eds). Child Phonology, Vol. 1: Production (1980), Academic Press, New York
89. Das S., Bakis R., Nadas A., Nahamoo D., and Picheny M. Influence of background noise and microphone on the performance of the IBM TANGORA speech recognition system. Proceedings of the IEEE International Conference on Acoustic Speech Signal Processing Vol. 2 (1993) 71-74
90. Gong Y. Speech recognition in noisy environments. Speech Communication 16 (1995) 261-291
91. Lockwood P., and Boudy J. Experiments with a non-linear spectral subtractor (NSS), Hidden Markov Models and the projection for robust speech recognition in cars. Speech Communication 11 (1992) 2-3
92. Lockwood P., and Boudy J. Experiments with a non-linear spectral subtractor (NSS), Hidden Markov Models and the projection for robust speech recognition in cars. Speech Communication 11 (1992) 215-228
93. Junqua J.C. The Lombard reflex and its role on human listeners and automatic speech recognizers. Journal of the Acoustical Society of America 93 1 (1993) 510-524
94. Lombard E. Le signe de l'elevation de la voix. Annals Maladiers Oreille, Larynx, Nez, Pharynx 37 (1911) 101-119
95. Hanley T.D., and Steer M.D. Effect of level of distracting noise upon speaking rate, duration and intensity. Journal of Speech and Hearing Disorders 14 (1949) 363-368
96. Schulman R. Articulatory dynamics of loud and normal speech. Journal of the Acoustical Society of America 85 (1989) 295-312
97. van Summers W.V., Pisoni D.B., Bernacki R.H., Pedlow R.I., and Stokes M.A. Effects of noise on speech production: Acoustic and perceptual analyses. Journal of the Acoustical Society of America 84 (1988) 917-928
98. Potash L.M. A signal detection problem and a possible solution in Japanese quail. Animal Behavior 20 (1972) 192-195
99. Sinott J.M., Stebbins W.C., and Moody D.B. Regulation of voice amplitude by the monkey. Journal of the Acoustical Society of America 58 (1975) 412-414
100. Siegel G.M., Pick H.L., Olsen M.G., and Sawin L. Auditory feedback in the regulation of vocal intensity of preschool children. Developmental Psychology 12 (1976) 255-261
101. Pick H.L., Siegel G.M., Fox P.W., Garber S.R., and Kearney J.K. Inhibiting the Lombard effect. Journal of the Acoustical Society of America 85 2 (1989) 894-900
102. Fuster-Duran A. Perception of conflicting audio-visual speech: An examination across Spanish and German. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 135-143
103. Massaro D.W. Bimodal speech perception: A progress report. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 79-101
104. Hennecke M.E., Stork D.G., and Prasad K.V. Visionary speech: Looking ahead to practical speechreading systems. In: Stork D.G., and Hennecke M.E. (Eds). Speechreading by Humans and Machines: Models, Systems and Applications. NATO ASI Series, Series F: Computer and Systems Sciences 150 (1996), Springer-Verlag, Berlin 331-349
105. Haton J.P. Automatic recognition in noisy speech. New Advances and Trends in Speech Recognition and Coding (1993), NATO Advanced Study Institute
106. Senior A., Neti C.V., and Maison B. On the use of visual information for improving audio-based speaker recognition. Proceedings of Auditory-Visual Speech Processing (AVSP) (1999) 108-111
107. Oviatt S.L. Multimodal signal processing in naturalistic noisy environments. In: Yuan B., Huang T., and Tang X. (Eds). Proceedings of the International Conference on Spoken Language Processing (ICSLP'2000) Vol. 2 (2000), Chinese Friendship Publishers, Beijing 696-699
108. Summerfield Q. Some preliminaries to a comprehensive account of audio-visual speech perception. In: Dodd B., and Campbell R. (Eds). Hearing by Eye: The Psychology of Lip-reading (1987), Lawrence Erlbaum, London 3-51
109. Petajan E.D. An improved automatic lipreading system to enhance speech recognition. Tech. Rep. 11251-871012-111TM (1987), AT&T Bell Labs
110. Iverson P., Bernstein L., and Auer E. Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition. Speech Communication 26 (1998) 1-2
111. Iverson P., Bernstein L., and Auer E. Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition. Speech Communication 26 (1998) 45-63
112. Oviatt S.L., Cohen P.R., and Wang M.Q. Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity. Speech Communication 15 (1994) 3-4
113. Oviatt S.L., Cohen P.R., and Wang M.Q. Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity. Speech Communication 15 (1994) 283-300
114. Oviatt S.L., DeAngeli A., and Kuhn K. Integration and synchronization of input modes during multimodal human-computer interaction. Proceedings of the Conference on Human Factors in Computing Systems (CHI'97) (1997), ACM Press, New York 415-422