Improvement of multimodal gesture and speech recognition performance using time intervals between gestures and accompanying speech

Eurasip Journal on Audio, Speech, and Music Processing

Volumn 2014, Issue , 2014, Pages

  Miki, Madoka ^a   Kitaoka, Norihide ^a   Miyajima, Chiyomi ^a   Nishino, Takanori ^b   Takeda, Kazuya ^a  

^a NAGOYA UNIVERSITY   (Japan)

^b MIE UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84894055880     PISSN: 16874714     EISSN: 16874722     Source Type: Journal    
DOI: 10.1186/1687-4722-2014-2     Document Type: Article

References (18)

1. Oviatt S, Coulston R, Lundsford R: When do we interact multimodally? Cognitive load and multimodal communication patterns. In Proceedings of ICMI. New York: ACM; 2004:129-136.
2. Dumas B, Signer B, Lalanne D: Fusion in multimodal interactive systems: an HMM-Based algorithm for user-induced adaptation. In Proceedings of 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York: ACM,; 2012:15-24.
3. Kettebekov S, Yeasin M, Sharma R: Prosody based co-analysis for continuous recognition of co-verbal gestures. In Proceedings of ICME. Washington DC: IEEE Computer Society; 2002:161-166.
4. Fukumoto M, Suenaga Y, Mase K: Finger-pointer: pointing interface by image processing. ACM Comput. Graph 1994,18(5):633-642.
5. Bolt RA: Put-that-there: voice and gesture at the graphics interface. ACM Comput. Graph 1980,14(3):262-270.
6. Hui P, Meng H: Joint interpretation of input speech and pen gestures for multimodal human computer interaction. In INTERSPEECH. Pittsburgh: ISCA,; 2006:1197-1200.
7. Qu S, Chai JY: Salience modeling based on non-verbal modalities for spoken language understanding. In Proceedings of ICMI. New York: ACM,; 2006:193-200.
8. Krahnstoever N, Kettebekov S, Yeasin M, Sharma R: A real-time framework for natural multimodal interaction with large screen displays. In Proceedings of ICMI. Piscataway: IEEE; 2002:349-354.
9. Lalanne D, Nigay L, Palanque P, Robinson P, Vanderdonckt J, Ladry J: Fusion engines for multimodal input: a survey. In Proceedings of ICMI-MLMI. New York: ACM; 2009:153-160.
10. Chai J, Hong P, Zhou M, Prasov Z: Optimization in multimodal interpretation. In Proceedings of ACL. Stroudsburg: Association for Computational Linguistics; 2004:1-8.
11. Johnston M: Finite-state multimodal parsing and understanding. In Proceedings of COLING. Stroudsburg: Association for Computational Linguistics; 2000:369-375.
12. Johnston M: Unification-based multimodal parsing. In Proceedings of COLING-ACL. Stroudsburg: Association for Computational Linguistics; 1998:624-630.
13. Wu L, Oviatt L, Cohen PR: Multimodal integration - a statistical view. Trans. Multimedia 1999,1(4):334-341.
14. Ban H, Miyajima C, Itou K, Takeda K, Itakura F: Speech recognition using synchronization between speech and finger tapping. In Proceedings of ICSLP. Pittsburgh: ISCA; 2004:943-946.
15. Shinoda K, Watanabe Y, Iwata K, Liang Y, Nakagawa R, Furui S: Semi-synchronous speech and pen input for mobile user interfaces. Speech Commun 2011,53(3):283-291.
16. Miki M, Miyajima C, Nishino T, Kitaoka N, Takeda K: An integrative recognition method for speech and gestures. In Proceedings of ICMI. New York: ACM; 2008:93-96.
17. Lee A, Kawahara T, Shikano K: Julius - an open source real-time large vocabulary recognition engine. In Proceedings of EUROSPEECH. Aalborg: ISCA; 2001:1691-1694.
18. Maekawa K: Corpus of spontaneous Japanese: its design and evaluation. In Proceedings of SSPR. Tokyo: ISCA and IEEE; 2003:7-12.