User-centered modeling and evaluation of multimodal interfaces

Proceedings of the IEEE

Volumn 91, Issue 9, 2003, Pages 1457-1468

  Oviatt, Sharon ^a  

^a OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

Evaluation; High fidelity simulations; Proactive interface design; Prototyping; Task analysis; User centered modeling

Indexed keywords

COMPUTER SIMULATION; ERROR ANALYSIS; EVALUATION; GESTURE RECOGNITION; HUMAN COMPUTER INTERACTION; INTERACTIVE COMPUTER SYSTEMS; MATHEMATICAL MODELS; SOFTWARE PROTOTYPING; SPEECH COMMUNICATION; SYNCHRONIZATION; WORLD WIDE WEB;

ERROR HANDLING; HIGH FIDELITY SIMULATIONS; MULTIMODAL SYSTEMS; PROACTIVE INTERFACE DESIGN; PROTOTYPING; TASK ANALYSIS; USER-CENTERED MODELING;

INTERFACES (COMPUTER);

EID: 21244476017     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2003.817127     Document Type: Conference Paper

References (88)

1. J. Benoit, C. Martin, C. Pelachaud, L. Schomaker, and B. Suhm, "Audio-visual and multimodal speech-based systems," in Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation, D. Gibbon, I. Mertins, and R. Moore, Eds. Boston, MA, Kluwer, 2000, pp. 102-203.
2. S. L. Oviatt, P. R. Cohen, L. Wu, J. Vergo, L. Dunean, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro, "Designing the user interface for multimodal speech and gesture applications; State-of-the-art systems and research directions," Human Comput. Interaction, vol. 15, no. 4, pp. 263-322, 2000.
3. P. R. Cohen, M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow, "Quickset: Multimodal interaction for distributed applications." in Proc. 5th ACM Int. Multimedia Conf., 1997, pp. 31-40.
4. X. Huang, A. Acero, C. Chelba, L. Deng, D. Duchene, J. Goodman, H. Hon, D. Jacoby, L. Jiang, R. Loynd, M. Mahajan, P. Mau, S. Meredith, S. Mughal, S. Neto, M. Plumpe, K. Wang, and Y. Wang, "MiPad: A next-generation PDA prototype," in Proc. ICSLP, vol. 3, 2000, pp. 33-36.
5. A. Adjoudani and C. Benoit, "Audio-visual speech recognition compared across two architectures," in Proc. Eurospeech, vol. 2, 1995, pp. 1563-1566.
6. C. Benoit and B. Le Goff, "Audio-visual speech synthesis from french text: Eight years of models, designs and evaluation," Speech Commun., vol. 26, pp. 117-129, 1998.
7. C. Bregler and Y. Konig, "Eigenlips for robust speech recognition," in Proc. ICASSP, vol. 2, 1994, pp. 669-672.
8. T. Chen. "Audiovisual speech processing." IEEE Signal Processing Mag., vol. 18, pp. 9-21, Jan. 2001.
9. S. Dupont and J. Lueitin. "Audio-visual speech modeling for continuous speech recognition." IEEE Trans. Multimedia, vol. 2, pp. 141-151, Sept. 2000.
10. E. D. Petajan, "Automatic lipreading to enhance speech recognition," Ph.D. dissertation, Univ. Illinois, Urbana-Champaign, 1984.
11. G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. Senior. "Recent advances in the automatic recognition of audio-visual speech," Proc. IEEE, vol. 91, pp. 1306-1326, Sept. 2003.
12. P. L. Silsbee and Q. Su, "Audiovisual sensory intergration using hidden Markov models," in Speechreading by Humana and Machines: Models, Systems and Applications, D. G. Stork and M. E. Hennecke, Eds. New York: Springer-Verlag, 1996, pp. 489-504.
13. D. G. Stork and M. E. Hennecke. Eds., Speechreading by Humans and Machines. New York: Springer-Verlag, 1996.
14. M. J. Tomlinson, M. J. Russell, and N. M. Brooke, "Integrating audio and visual information to provide highly robust speech recognition," in Proc: ICASSP, vol. 2, 1996, pp. 821-824.
15. P. Rubin, E. Vatikiotis-Bateson, and C. Benoit, Eds., "Audio-visual speech processing," in Speech Commun. (Special Issue), 1998, vol. 26.
16. L. Duncan, W. Brown, C. Esposito, H. Holmback, and P. Xue, "Enhancing virtual maintenance environments with speech understanding." Boeing M&CT TechNet, 1999.
17. R. Sharma, V. I. Pavlovic, and T. S. Huang, "Toward multimodal human-computer interface," Proc. IEEE, vol. 86, pp. 853-860, May 1998.
18. N. Negroponte, "Report for ONR and DARPA." Architecture Machine Group, Massachusetts Inst. Technol., Cambridge, 1978.
19. V. Pavlovic, R. Sharma, and T. Huang, "Visual interpretation of hand gestures for human-computer interaction: A review," IEEE Trans. Pattern Anal. Machine Intell., vol. 19, pp. 677-695, July 1997.
20. S. Zhai, C. Morimoto, and S. Ihde, "Manual and gaze input cascaded (MAGIC) pointing," in Proc. Conf Human Factors Computing Systems (CHI'99), 1999, pp. 246-253.
21. J. Bers, S. Miller, and J. Makhoul, "Designing conversational interfaces with multimodal interaction." in Proc. DARPA Workshop Broadcast News Understanding Systems, 1998, pp. 319-321.
22. A. Cheyer. "MVIEWS: Multimodal tools for the video analyst," in Proc. Int. Conf. Intelligent User Interfaces (IUI'98), 1998, pp. 55-62.
23. T. G. Holzman, "Computer-human interface solutions for emergency medical care." Interactions, vol. 6, no. 3, pp. 13-24, 1999.
24. C. Neti, G. Iyengar, G. Putamianos, and A. Senior, "Perceptual interfaces for information interaction: Joint processing of audio and visual information for human-computer interaction," in Proc: ICSLP, vol. 3, 2000, pp. 11-14.
25. S. Pankanti, R. M. Bolle, and A. Jain, "Guest editors' introduction: Biometrics-the future of identification," IEEE Computer, vol. 33, pp. 46-80, Feb. 2000.
26. O. Slock, "Language-based interfaces and their application for cultural tourism," AI Mag., pp. 85-97, 2001.
27. W. Wahlster, "SmartKom: Multimodal dialogs with mobile Web users," in Proc. Cyber Assist Int. Symp., International Forum, 2001, pp. 33-34.
28. A. Waibel, B. Suhm, M. T. Vo, and J. Yang, "Multimodal interfaces for multimedia information agents." in Proc. ICASSP, vol. 1, 1997, pp. 167-170.
29. S. L. Oviatt and P. R. Cohen, "Multimodal systems that process what comes naturally," Commun. ACM, vol. 43, no. 3, pp. 45-53, Mar. 2000.
30. S. L. Oviatt, P. R. Cohen, M. W. Fong, and M. P. Frank, "A rapid semi-automatic simulation technique for investigating interactive speech and handwriting." in Proc. ICSLP, vol. 2, 1992, pp. 1351-1354.
31. W. S. Condon, "An analysis of behavioral organization," Sign Lang. Stud., vol. 58, pp. 55-88, 1988.
32. A. Kendon, "Gesticulation and speech: Two aspects of the process of utterance," in The Relationship of Verbal and Nonverbal Communication, M. Key, Ed. The Hague, The Netherlands: Mouton, 1980, pp. 207-227.
33. D. McNeill, Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
34. F. Quek, Y. Xiong, and D. McNeill, "Gestural trajectory symmetries and discourse segmentation." in Proc. ICSLP, vol. 1, 2002, pp. 185-188.
35. F. Quek, D. McNeill, R. Bryll, and M. Harper, "Gestural spatialization in natural discourse segmentation." in Proc. ICSLP, vol. 1, 2002, pp. 189-192.
36. D. W. Massaro and D. G. Stork, "Sensory integration and specchreading by humans and machines," Amer. Scientist, vol. 86, pp. 236-244, 1998.
37. M. McGrath and Q. Summerfield, "Intermodal timing relations and audio-visual speech recognition by normal-hearing adults," J. Acoust. Soc. Amer., vol. 77. no. 2. pp. 678-685, 1985.
38. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, 1976.
39. J. Robert-Ribes, J.-L. Schwartz, T. Lallouache, and P. Escudier, "Complementarity and synergy in bimodal speech: Auditory, visual, and auditory-visual identification of French oral vowels in noise." J. Acoust. Soc. Amer., vol. 103, no. 6, pp. 3677-3689, 1998.
40. M. Turk and G. Robertson, Eds., "Perceptual user interfaces," in Commun. ACM, 2000, vol. 43, pp. 32-70.
41. B. Fröba, C. Rothe, and C. Küblbeck, "Statistical sensor calibration for fusion of different classifiers in a biometric person recognition framework," in Lecture Notes in Computer Science, Multiple Classifier Systems Heidelberg, Germany, 2000, vol. 1857, pp. 362-371.
42. A. Jain, L. Hong, and Y. Kulkarni, "A multimodal biometric system using fingerprint, face and speech." in Proc. 2nd Int. Conf. Audio- and Video-Based Biometric Person Authentication, 1999. pp. 182-187.
43. A. Jain and A. Ross, "Learning user-specific parameters in a multibiometric system," presented at the Int. Conf. Image Processing (ICIP), Rochester, NY, 2002.
44. A. Ross, A. Jain, and J. Z. Qian, "Information fusion in biometrics," in Lecture Notes in Computer Science, Audio- and Video-Based Biometric Person Authentication Heidelberg, Germany, 2001, vol. 2091, pp. 354-359.
45. S. L. Oviatt, A. DeAngeli, and K. Kuhn, "Integration and synchronization of input modes during multimodal human-computer interaction." in Proc. Conf. Human Factors Computing Systems (CHI'97), 1997, pp. 415-422.
46. N. Dahlbäck, A. Jëonsson, and L. Ahrenberg, "Wizard of Oz studies - why and how." in Proc. Int. Workshop Intelligent User Interfaces, 1992, pp. 193-200.
47. S. L. Oviatt, R. Collision, S. Shriver, B. Xiao, R. Wesson, R. Lunsford, and L. Carmichael, "Toward a theory of organized multimodal integration patterns during human-computer interaction," presented at the Int. Conf. Multimodal Interfaces, Vancouver, BC, Canada, 2003.
48. W. J. M Levelt, Speaking: From Intentions to Articulation. Cambridge, MA: MIT Press, 1989.
49. M. A. Just and P. A. Carpenter, "Eye fixations and cognitive processes," Cogn. Psychol., vol. 8, pp. 441-480, 1976.
50. K. Rayner, "Eye movements in reading and information processing: Twenty years of research," Psychol. Bull., vol. 124, no. 3, pp. 372-422, 1998.
51. R. J. K. Jacob, "The use of eye movements in human-computer interaction techniques: What you look at is what you get," ACM Trans Inform. Syst., vol. 9, pp. 152-169, 1991.
52. P. P. Maglio, R. Barrett, C. S. Campbell, and T. T. Selker, "SUITOR: An attentive information system." in Proc. Int. Conf. Intelligent User Interfaces (IUI 2000), 2000, pp. 169-176.
53. A. G. Hauptmann, "Speech and gestures for graphic image manipulation." in Proc. Conf. Human Factors Computing Systems (CHI'89), 1989, pp.241-245.
54. S. L. Oviatt, "Multimodal interactive maps: Designing for human performance." Human Comput. Interaction, vol. 12, no. 1-2, pp. 93-129, 1997.
55. P. Cohen and S. L. Oviatt, "The role of voice input for human-machine communication," Proc. Nat. Acad. Sci., vol. 92, pp. 9921-9927, 1995.
56. S. L. Oviatt and P. R. Cohen, "Discourse structure and performance efficiency in interactive and noninteractive spoken modalities." Comput. Speech Lang., vol. 5, no. 4, pp. 297-326, 1991.
57. S. L. Oviatt and E. Olsen, "Integration themes in multimodal human-computer interaction." in Proc. ICSLP, vol. 2, 1994, pp. 551-554.
58. B. Suhm, "Multimodal interactive error recovery for non-conversational speech user interfaces," Ph.D. dissertation. Fredericiana University, Karlsruhe, Germany, 1998.
59. R. A. Bolt, "Put-that-there: Voice and gesture at the graphics interface," Comput. Graph., vol. 14, no. 3, pp. 262-270, 1980.
60. D. Koons, C. Sparrell, and K. Thorisson, "Integrating simultaneous input from speech, gaze, and hand gestures," in Intelligent Multimedia Interfaces, M. Maybury, Ed. Cambridge, MA: MIT Press, 1993, pp. 257-276.
61. B. Xiao, C. Girand, and S. L. Oviatt, "Multimodal integration patterns in children." in Proc. ICSLP, 2002. pp. 629-632.
62. B. Xiao, R. Lunsford, R. Coulston, M. Wesson, and S. L. Oviatt, "Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences," presented at the Int. Conf. Multimodal Interfaces, Vancouver, BC, Canada, 2003.
63. K. Naughton, "Spontaneous gesture and sign: A study of ASL signs co-occurring with speech," in Proc. Workshop Integration Gesture Language and Speech, 1996, pp. 125-134.
64. C. Abry, M. T. Lallouache, and M. A. Cathiard, "How can coarticulation models account for speech sensitivity to audio-visual de synchronization?." in Speechreading by Humans and Machines: Models, Systemsand Applications, D. G. Stork and M. E. Hennecke, Eds. New York: Springer-Verlag, 1996, pp. 247-255.
65. A. Kendon. "Some functions of gaze direction in social interaction," Acta Psychol., vol. 26, pp. 22-63, 1967.
66. Z. M. Griffin and K. Bock, "What the eyes say about speaking." Psychol. Sci., vol. 11, no. 4, pp. 274-279, 2000.
67. S. L. Oviatt, "Ten myths of multi modal interaction." Commun. ACM, vol. 42, no. 11, pp. 74-81. Nov. 1999.
68. Proc. 3rd International ACM Proceedings of the Conf. Assistive Technologies (ASSETS'98), A. I. Karshmer and M. Blattner. Eds., Marina del Rey, CA, 1998.
69. R. Markinson, private communication, 1993.
70. P. B. Kricos, "Differences in visual intelligibility across talkers," in Speechreading by Humans and Machines: Models, Systems and Applications, D. G. Stork and M. E. Hennecke, Eds. New York: Springer-Verlag, 1996, pp. 43-53.
71. K. Sekiyama, Y. Tohkura, and Y. McGurk, "Effect in nonenglish listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility," J. Acoust. Soc. Amer., vol. 90, pp. 1797-1805, 1991.
72. A. Fuster-Duran, "Perception of conflicting audio-visual speech: An examination across Spanish and German," in Speechreading by Humans and Machines: Models, Systems and Applications, D. G. Stork and M. H. Hennecke, Eds. New York: Springer-Verlag, 1996, pp. 135-143.
73. M. Argyle, "Nonverbal communication in human social interaction," in Nonverbal Communication. R. Hinde, Ed. Cambridge, MA: Cambridge Univ. Press, 1972, pp. 243-267.
74. C. D. Wickens, D. L. Sandry, and M. Vidulich, "Compatibility and resource competition between modalities of input, central processing, and output," Human Factors, vol. 25, pp. 227-248, 1983.
75. P. Cohen, M. Dalrymple, D. Moran, and F. Pereira, "Synergistic use of direct manipulation and natural language," in Proc. Conf. Human Factors Computing Systems (CHI'89), 1989, pp. 227-234.
76. S. L. Oviatt, P. R. Cohen, P. R., and M. Q. Wang, "Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity," Speech Commun., vol. 15, pp. 283-300, 1994.
77. J. H. Leatherby and R. Pausch, "Voice input as a replacement for keyboard accelerators in a mouse-based graphical editor: An empirical study," J. Amer. Voice Input/Output Soc., vol. 11, no. 2, July 1992.
78. P. R. Cohen, D. R. McGee, and J. Clow. "The efficiency of multimodal interaction for a map-based task," in Proc. Language Technology-Joint Conf. (ANLP-NAACL 2000), 2000, pp. 331-338.
79. S. L. Oviatt and K. Kuhn, "Referential features and linguistic indirection in multimodal language," in Proc. ICSLP. vol. 2, 1998, pp. 227-280.
80. S. L. Oviatt. "Mutual disambiguation of recognition errors in a multimodal architecture," in Proc. Conf. Human Factors Computing Systems (CHI'99), 1999, pp. 576-583.
81. A. Rudnicky and A. Hauptman. "Multimodal interactions in speech systems," in Multimedia Interface Design, M. Blattner and R. Dannenberg, Eds. New York: ACM. 1992, Frontier Series, pp. 147-172.
82. S. L. Oviatt, "Breaking the robustness barrier: Recent progress on the design of robust multimodal systems," in Advances in Computers, M. Zelkowitz, Ed. New York: Academic, 2002, vol. 56, pp. 305-341.
83. S. L. Oviatt, J. Bernard, and G. Levow, "Linguistic adaptations during spoken and multimodal error resolution," Lang. Speech, vol. 41, no. 3-4, pp. 515-438, 1999.
84. A. McLeod and Q. Summerfield, "Quantifying the contribution of vision to speech perception in noise," Br. J. Audiol., vol. 21, pp. 131-141, 1987.
85. P. Iverson, L. Bernstein, and E. Auer, "Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recognition," Speech Commun., vol. 26, no. 1-2, pp. 45-63, 1998.
86. A. Senior, C. Neti, and B. Maison. "On the use of visual information for improving audio-based speaker recognition." in Proc. AuditoryVisual Speech Processing (AVSP) 1999, pp. 108-111.
87. S. Nakamura, H. Ito, and K. Shikano, "Stream weight optimization of speech and lip image sequence for audio-visual speech recognition," in Proc. ICSLP, vol. 3, 2000, pp. 20-24.
88. G. Potamianos and C. Neti, "Stream confidence estimation for audio-visual speech recognition," in Proc. ICSLP. vol. 3, 2000, pp. 746-749.