Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures

ICMI'04 - Sixth International Conference on Multimodal Interfaces

Volumn , Issue , 2004, Pages 175-182

  Holzapfel, Hartwig ^a   Nickel, Kai ^a   Stiefelhagen, Rainer ^a  

^a UNIVERSITY OF KARLSRUHE   (Germany)

Author keywords

Gesture; Multimodal architectures; Multimodal fusion and multisensory integration; Natural language; Speech; Vision

Indexed keywords

GESTURE RECOGNITION; HUMAN ENGINEERING; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; SPEECH RECOGNITION; USER INTERFACES;

MULTIMODAL ARCHITECTURES; MULTIMODAL FUSION AND MULTISENSORY INTEGRATION; SPEECH, VISION, NATURAL LANGUAGE, GESTURE; TEMPORAL CORRELATION;

HUMAN COMPUTER INTERACTION;

EID: 14944340532     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1027933.1027964     Document Type: Conference Paper

References (27)

1. T. Asfour, A. Ude, K.Berns, and R. Dillmann. Control of armar for the realization of anthropomorphic motion patterns. In The second IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS 2001), pages 22-24, 2001.
2. R. A. Bolt, "put-that-there": Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer Graphics and Interactive Techniques, pages 262-270. ACM Press, 1980.
3. B. Carpenter. The Logic of Typed Feature Structures. Cambridge University Press, 1992.
4. P. R. Cohen, R. Coulston, and K. Krout. Multimodal interaction during multiparty dialogues: Initial results. In Proceedings of the International Conference On Multimodal Interfaces, 2002.
5. A. Corradini, R. M. Wesson, and P. R. Cohen. A map-based system using speech and 3d gestures for pervasive computing. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, 2002.
6. M. Denecke. Object-oriented techniques in grammar and ontology specification. In The Workshop on Multilingual Speech Communication, pages 59-64, Kyoto, Japan, 2000.
7. M. Denecke. Rapid prototyping for spoken dialogue systems. In Proceedings of the 19th International Conference on Computational Linguistics, Taiwan, 2002.
8. J. Eisenstein and C. M. Christoudias. A salience-based approach to gesture-speech alignment. In Proceedings of the Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting, 2004.
9. M. Finke, P. Geutner, H. Hild, T. Kemp, K. Ries, and M. Westphal. The karlsruhe-verbmobil speech recognition engine. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, ICASSP-97, Munich, Germany, 1997.
10. C. Fuegen, H. Holzapfel, and A. Waibel. Tight coupling of speech recognition and dialog management dialog-context dependent grammar weighting for speech recognition. In Proceedings of the International Conference on Spoken Language Processing, 2004.
11. P. Gieselmann, C. Fuegen, H. Holzapfel, T. Schaaf, and A. Waibel. Towards multimodal communication with a household robot. In Proceedings of the International Conference on Humanoid Robots, 2003.
12. Third IEEE International Conference on Humanoid Robots - Humanoids, Karlsruhe, Germany, 2003.
13. IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, 2004.
14. M. Johnston. Unification-based multimodal parsing. In COLING-ACL, pages 624-630, 1998.
15. M. Johnston, S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, and P. Maloor. Match: An architecture for multimodal dialogue systems. In Proceedings of ACL, 2002.
16. M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, and I. Smith. Unification-based multimodal integration. In Proceedings of the Thirty-Fifth Annual Meeting of the Association for Computational Linguistics, pages 281-288, 1997.
17. M. Kaur, M. Tremaine, N. Huang, J. Wilder, Z. Gacovski, F. Flippo, and C. S. Mantravadi. Where is "it"? event synchronization in gaze-speech input systems. In Proceedings of the 5th International Conference on Multimodal Interfaces, pages 151-158. ACM Press, 2003.
18. S. Kettebekov, M. Yeasin, and R. Sharma. Prosody based co-analysis for continuous recognition of coverbal gestures. In Proceedings of the 4th International Conference on Multimodal Interfaces, Pittsburgh, USA, 2002.
19. S. C. Levinson. Pragmatics. Cambridge, England: Cambridge University, 1983.
20. K. Nickel and R. Stiefelhagen. Pointing gesture recognition based on 3d-tracking of face, hands and head orientation. In Proceedings of the Fifth International Conference on Multimodal Interfaces, Vancouver, Canada, Nov. 5-7 2003.
21. S. L. Oviatt, A. DeAngeli, and K. Kuhn. Integration and synchronization of input modes during multimodal human-computer interaction. In CHI, pages 415-422, 1997.
22. N. Reithinger, J. Alexandersson, T. Becker, A. Blocher, R. Engel, M. Lckelt, J. Mller, N. Pfleger, P. Poller, M. Streit, and V. Tschernomas. Smartkom - adaptive and flexible multimodal access to multiple applications. In Proceedings of the Fifth International Conference on Multimodal Interfaces, 2003.
23. R. Sharma, V. Pavlovic, and T. Huang. Toward multimodal human-computer interface. In Proceedings of the IEEE, volume 86, pages 853 - 869, May 1998.
24. H. Soltau, F. Metze, C. Fuegen, and A. Waibel. A one pass- decoder based on polymorphic linguistic context assignment. In Proceedings of the Automatic Speech Recognition and Understanding Workshop, ASRU-8001, Madonna di Campiglio, Trento, Italy, December 2001.
25. R. Stiefelhagen, C. Fügen, P. Gieselmann, H. Holzapfel, K. Nickel, and A. Waibel. Natural human-robot interaction using speech, gaze and gestures. In Proceedings of the International Conference on Intelligent Robots and Systems, Sendai, Japan, 2004.
26. I. Wachsmuth. Communicative rhythm in gesture and speech. In Gesture-Based Communication in Human-Computer Interaction: International Gesture Workshop, GW'99, Gif-sur-Yvette, France, March 1999.
27. L. Wu, S. L. Oviatt, and P. R. Cohen. Multimodal integration - a statistical view. IEEE Transactions on Multimedia, 1(4):334-341, 1999.