Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions

Human-Computer Interaction

Volumn 15, Issue 4, 2000, Pages 263-322

  Oviatt, S ^a   Cohen, P ^a   Wu, L ^a   Vergo, J ^a   Duncan, L ^a   Suhm, B ^a   Bers, J ^a   Holzman, T ^a   Winograd, T ^a   Landay, J ^a   Larson, J ^a   Ferro, D ^a  

^a OREGON HEALTH AND SCIENCE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COGNITIVE SYSTEMS; COMPUTER SIMULATION; ERROR ANALYSIS; HUMAN COMPUTER INTERACTION; NATURAL LANGUAGE PROCESSING SYSTEMS; SOFTWARE ENGINEERING; SPEECH PROCESSING; STATISTICAL METHODS; VIRTUAL REALITY; WORLD WIDE WEB;

MULTIMODAL INTERFACES;

GRAPHICAL USER INTERFACES;

EID: 0034448810     PISSN: 07370024     EISSN: None     Source Type: Journal    
DOI: 10.1207/S15327051HCI1504_1     Document Type: Article

References (141)

1. On the integration of auditory and visual parameters in an HMM-based ASR
2. Towards a general theory of action and time
3. Analyzing intention in dialogues
4. Employing AI methods to control the behav- ior of animated interface agents
5. Audio-visual and multimodal speech systems
6. Designing conversational interfaces with multimodal interaction
7. Put-that-there: Voice and gesture at the graphics interface
8. Improving connected letter recognition by lipreading
9. A rational agent as the kernel of a cooperative spoken dialogue system: Implementing a logical theory of interaction
10. Typed unification for natural language processing
11. Typed feature structures: Inheritance, (in)equality, and extensionality
12. Living hand to mouth: Psychological theories about speech and gesture in interactive dialogue systems
13. Multimodal maps: An agent-based approach
14. A unified framework for constructing multimodal experiments and applications
15. STAMP: A suite of tools for analyzing multimodal system processing
16. Interactive simulation in a multi-person virtual world
17. Design principles for intelligent environments
18. An open agent architecture
19. Synergistic use of direct manipulation and natural language
20. The efficiency of multimodal interaction: A case study
21. Quickset: Multimodal interaction for distributed applications
22. Rational interaction as the basis for communication
23. Multimodal interaction for 2D and 3D environments
24. The role of voice input for human-machine communication
25. Elements of a plan-based theory of speech acts
26. The challenge of spoken language systems: Research directions for the nineties
27. Finding structure in time
28. A multi-level organization for problem solving using many diverse sources of knowledge
29. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment
30. COMET: Generating coordinated multi-media explanations
31. Using the baby-babble-blanket for infants with motor problems
32. Auto-directive microphone systems
33. Finger-pointer: Pointing interface by image processing
34. The role of the DARPA communicator architecture as a human computer interface for distributed simulations
35. Speech recognition in noisy environments: A survey
36. Speech and gestures for graphic image manipulation
37. A word sense checking application for Simplified English
38. A pragmatic principle for agent communication
39. Computer-human interface solutions for emergency medical care
40. Principles of mixed-initiative user interfaces
41. Microsoft Windows highly intelligent speech recognizer: Whisper
42. Modeling the interaction of phonemic intelligibility and lexical structure in audiovisual word recogniton
43. Unification-bazed multimodal parsing
44. Unification-bazed multimodal integration
45. Discriminative learning for minimum error classification
46. Patterns of entry and correction in large vocabulary continuous speech recognition systems
47. Discriminative training-recent progress in speech recognition
48. Functional grammar
49. An iterative design methodology for user-friendly natural language office information applications
50. Gesticulation and speech: Two aspects of the process of utterance
51. SUEDE: A Wizard of Oz prototyping tool for speech user interfaces
52. Towards a fault-tolerant multi-agent system architecture
53. MedSpeak: Report creation with continuous speech recognition
54. Interactive sketching for the early stages of user interface design
55. Sketching storyboards to illustrate interface behavior
56. Designing the user interface for pen and speech applications
57. Deictic believability: Coordinated gesture, locomotion, and speech in lifelike pedagogical agents
58. Pointing and voicing in deictic expressions
59. System descriptions and performance summary
60. The open agent architecture: A framework for building distributed software systems
61. Creating tangible interfaces by augmenting physical objects with mutilmodal language
62. Confirmation in multimodal systems
63. Something from nothing: Augmenting a paper-based work practice with multimodal interaction
64. Hearing lips and seeing voices
65. Adaptive bimodal sensor fusion for automatic speechreading
66. A framework for representing knowledge
67. Intelligent multimedia interface technology
68. A natural language processing based situation display
69. Pen/voice: Complementary multimodal communication
70. User-centered design of spoken language and multimodal interfaces
71. Multimodal interactive maps: Designing for human performance
72. Mutual disambiguation of recognition errors in a multimodal architecture
73. Ten myths of multimodal interaction
74. Multimodal system processing in mobile environments
75. Taming recognition errors with a multimodal interface
76. Linguistic adaptation during error resolution with spoken and multimodal systems
77. Discourse structure and performance efficiency in interactive and noninteractive spoken modalities
78. Multimodal systems that process what comes naturally
79. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting
80. Toward interface design for human language technology: Modality and structure as determinants of linguistic complexity
81. Integration and synchronization of input modes during multimodal human-computer interaction
82. Referential features and linguistic indirection in multimodal language
83. Integration themes in multimodal human-computer interaction
84. Interacting with animated characters: Research infrastructure and next-generation interface design
85. Error resolution during multimodal human-computer interaction
86. Biometrics: The future of identification
87. Feature-based language understanding
88. Integration of audio/visual information for use in human-computer intelligent interaction
89. Multimodal prediction and classification on audio-visual features
90. Smart rooms
91. An application of default logic to speech act theory
92. A tutorial on hidden Markov models and selected applications in speech recognition
93. Dialogue act classification using language models
94. Recognition-based user interfaces
95. Adaptive fusion of acoustic and visual sources for automatic speech recognition
96. Graphics and natural language as components of automatic explanation
97. Special issue on audio-visual speech processing
98. Multimodal interactions in speech systems
99. Dialogue acts are rational plans
100. Combining visual and acoustic speech signal with a neural network improves intelligibility
101. Galaxy-II: A reference architecture for conversational system development
102. An architecture for multimodal information fusion
103. Interactive voice technology for telecommunications applications (IVTTA'96)
104. Interactive recovery from speech recognition errors in speech user interfaces
105. The CSLU Toolkit: Rapid prototyping of spoken language systems
106. Perceptual user interfaces
107. A statistical approach to multimodal natural language interaction
108. Multimodal learning interfaces
109. Building an application framework for speech and pen input integration in multimodal learning interfaces
110. Prototyping: A day in the life of an interface designer
111. Plan-based integration of natural language and graphics generation
112. Phenome recognition using time-delay neural networks
113. Multimodal interfaces
114. Integration of eye-gaze, voice and manual response in multimodal user interfaces
115. Compatibility and resource competition between modalities of input, central processing, and output
116. Principles of S-R-C compatibility with spatial and verbal tasks: The role of display-control interfacing
117. Getting a controlled language off the ground at Boeing
118. The use of hand-drawn gestures for text editing
119. Multimodal integration - A statistical view
120. Manual and gaze input cascaded (MAGIC) pointing