Multimodal model integration for sentence unit detection

ICMI'04 - Sixth International Conference on Multimodal Interfaces

Volumn , Issue , 2004, Pages 121-128

  Chen, Lei ^a   Harper, Mary P ^a   Liu, Yang ^b,d   Shriberg, Elizabeth ^b,c  

^a Purdue University   (United States)

^b INTERNATIONAL COMPUTER SCIENCE INSTITUTE   (United States)

^c SRI INTERNATIONAL   (United States)

^d Elec and Comp Eng at Purdue ^*  (United States)

Author keywords

Dialog; Gesture; Language models; Multimodal fusion; Prosody; Sentence boundary detection

Indexed keywords

DIALOG; LANGUAGE MODELS; MULTIMODAL FUSION; PROSODY; SENTENCE BOUNDARY DETECTION;

COMPUTER SIMULATION; HUMAN COMPUTER INTERACTION; IMAGE PROCESSING; LINGUISTICS; SPEECH COMMUNICATION; VIDEO RECORDING; VISUAL COMMUNICATION;

GESTURE RECOGNITION;

EID: 14944345809     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. http://vislab.cs.wright.edu/kdi/.
2. http://www.darpa.mil/ipto/programs/ears/.
3. P. Boersma and D. Weeninck. Praat, a system for doing phonetics by computer. Technical Report 132, University of Amsterdam, Inst. of Phonetic Sc., 1996.
4. T. Brants. TnT-a statistical part-of-speech tagger. In ANLP, pages 224-231, 2000.
5. P. F. Brown, V. J. D. Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer. Class-based n-gram models of natural language. Computational Linguistics, pages 467-479, 1992.
6. R. Bryll, F. Quek, and A. Esposito. Automatic hand hold detection in natural conversation. In IEEE Workshop on Cues in Communication, Kauai,Hawaii, Dec 2001.
7. W. Buntine. Learning classification trees. Statistics and Computing, 2:63-73, 1992.
8. J. Cassell and M. Stone. Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems. In AAAI, 1999.
9. J. Chai, P. Hong, and M. Zhou. A probabilistic approach to reference resolution in multimodal user interfaces. In Proc. of the 2004 International Conference on Intelligent User Interfaces (IUI-04), Madeira, Portugal, January 2004.
10. L. Chen, M. P. Harper, and F. Quek. Gesture patterns during speech repairs. In Proc. of the Fourth IEEE International Conference on Multimodal Interfaces(ICMI'02), Pittsburg, PA, Oct. 2002.
11. S. Coquoz. Broadcast news segmentation using mde and stt information to improve s peech recognition. Technical report, International Computer Science Institute, 2004.
12. A. Esposito, K. E. McCullough, and F. Quek. Disfluencies in gesture: Gestural correlates to speech silent and filled pauses. In Proceeding of IEEE Workshop on Cues in Communication, Kauai,Hawaii, 2001.
13. S. Fels and G. Hinton. Glove-talk II - A neural-network interface which maps gestures to parallel formant speech synthesizer controls. IEEE Transactions on Neural Networks, 8:977-984, Sept. 1997.
14. F.Quek, D. McNeill, R. Ansari, X. Ma, R. Bryll, S. Duncan, K.-E. McCullough, and C. Kirbas. Gestures cues for conversational interaction in monocular video. In ICCV'99 Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pages 119-126, Corfu, Greece, 1999.
15. D. Gibbon, B. Hell, K. Looks, and T. Trippel. Formal syntax of gesture : Cogest1.1. Technical report, Univ. of Bielefield, 2003.
16. A. Kendon. Some relationships between body motion and speech. In A. W. Siegman and B. Pope, editors, Studies in Dynamic Communication. Pergamon, New York, 1972.
17. S. Kettebekov, M. Yeasin, and R. Sharma. Prosody based co-analysis for continuopus recognition of coverbal gestures. In International Conference on Multimodal Interfaces (ICMI'02), Pittsburgh USA, 2002.
18. Y. Liu, A. Stolcke, E. Shriberg, and M. P. Harper. Comparing and combining generative and posterior probability models: Some advances in sentence boundary detection in speech. In Proceedings of the Empirical Methods in Natural Language Processing, 2004.
19. Y. Liu, A. Stolcke, E. Shriberg, and M. P. Harper. Using machine learning to cope with imbalanced classes in natural speech: Evidence from sentence boundary and disfluency detection. In Proceedings of the International Conference on Spoken Language Processing, 2004.
20. M. Mateer and A. Taylor. Disfluency annotation Stylebook for the Switchboard corpus. Technical report, Department of Computer and Information Science, University of Pennsylvania, 1995.
21. D. McNeil. Hand and Mind: What Gestures Reveal about Thought. Univ. Chicago Press, 1992.
22. D. McNeill and S. Duncan, Growth points in thinking-for-speaking, chapter 7, pages 141-161. Cambridge Univ. Press, 2000.
23. F. Quek, R. Bryll, and X. Ma. A parallel algorighm for dynamic gesture tracking. In ICCV Workshop on RATFG-RTS, Gorfu.Greece, 1999.
24. F. Quek, M. P. Harper, Y. Haciahmetoglu, L. Chen, and L. Ramig. Speech pauses and gestural holds in Parkinson's disease. In Seventh International Conference on Spoken Language Processing, ICSLP, Denver,CO, Sept. 2002.
25. F. Quek, D. McNeill, R. Bryll, S. Duncan, X. Ma, C. Kirbas, K. E. Mccullough, and R. Ansari. Multimodal human discourse: Gesture and speech. ACM Transactions on Computer-Human Interaction, 9(3), Sept. 2002.
26. F. Quek, Y. Shi, C. Kirbas, and S. Wu. Vissta: A tool for analyzing multimodal discourse data. In Seventh International Conference on Spoken Language Processing, Denver,CO, Sept. 2002.
27. F. Quek and Y. Xiong. Oscillatory gestures and discourse. In Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP2003, Hong Kong, April 2003.
28. F. Quek, Y. Xiong, and D. McNeill. Gestural trajectory symmetries and discourse segmentation. In 7th ICSLP, Denver, CO, Sept. 2002.
29. L. Rabiner and B. Juang. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1):4-16, 1986.
30. R. Sharma, J. Cai, S. Chakravarthy, I. Poddar, and Y. Sethi. Exploiting speech/gesture co-occurrence for improving continuous gesture recognition in weather narration. In Proceedings of Automatic Face and Gesture Recognition, 2000.
31. E. Shriberg and A. Stolcke. Direct modeling of prosody: An overview of applications in automatic speech processing. In International Conference on Speech Prosody, 2004.
32. E. Shriberg, A. Stolcke, D. Hakkani-Tur, and G. Tur. Prosody-based automatic segmentation of speech into sentences and topics. Speech Communications, pages 127-154, 2000.
33. K. Sonmez, E. Shriberg, L. Heck, and M. Weintraub. Modeling dynamic prosodie variation for speaker verification. In Proceedings of the International Conference on Spoken Language Processing, pages 3189-3192, 1998.
34. A. Stolcke and E. Shriberg. Statistical language modeling for speech disfluencies. In ICASSP, 1996.
35. S. Strassel. Simple Metadata Annotation Specification. Linguistic Data Consortium, 5.0 edition, 2003.
36. A. Wilson, A. Bobick, and J. Cassell. Recovering the temporal structure of natural gesture. In Automatic Face and Gesture Recognition, pages 66-71, Oct. 1996.