A salience driven approach to robust input interpretation in multimodal conversational systems

HLT/EMNLP 2005 - Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2005, Pages 217-224

  Chai, Joyce Y ^a   Qu, Shaolin ^a  

^a Michigan State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONCEPT IDENTIFICATION; CONVERSATIONAL SYSTEMS; GRAPHICAL DISPLAYS; MULTIMODAL CONVERSATION; MULTIMODAL INPUTS; PHYSICAL WORLD; SPOKEN LANGUAGE UNDERSTANDING; WORD ERROR RATE;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 80053238919     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/1220575.1220603     Document Type: Conference Paper

References (34)

1. Bangalore, S. and Johnston, M. 2000. Integrating Multimodal Language Processing with Speech Recognition. In Proceedings of ICSLP.
2. Brown, P., Della Pietra, V. J., deSouza, P. V., Lai, J. C, and Mercer, R. L. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467-479.
3. Byron, D., Mampilly, T., Sharma, V., and Xu, T. 2005. Utilizing Visual Attention for Cross-Modal Coreference Interpretation. Spring Lecture Notes in Computer Science: Proceedings of Context-05, page 83-96.
4. Cassell, J., Stone, M., Douville, B., Prevost, S., Achorn, B., Steedman, M., Badler, N., and Pelachaud, C. 1994. Modeling the interaction between speech and gesture. Cognitive Science Society.
5. Chai, J. Y., Prasov, Z., Blaim, J., and Jin, R. 2005. Linguistic Theories in Efficient Multimodal Reference Resolution: an Empirical Investigation. The 10th International Conference on Intelligent User Interfaces (IUI-05), pp. 43-50, San Diego, CA.
6. Chai, J. Y., Hong, P., Zhou, M. X, and Prasov, Z. 2004b. Optimization in Multimodal Interpretation. In Proceedings of ACL, pp. 1-8, Barcelona, Spain.
7. Chai, J. Y., Hong, P., and Zhou, M. 2004a. A Probabilistic Approach to Reference Resolution in Multimodal User Interfaces. Proceedings of 9th International Conference on Intelligent User Interfaces (IUI-04), pp. 70-77, Madeira, Portugal. (Pubitemid 40673704)
8. Chelba, C. and Jelinek, F. 2000. Structured language modeling. Computer Speech and Language, 14(4):283-332.
9. Cohen, P., Johnston, M., McGee, D., Oviatt, S., Pittman, J.; Smith, I., Chen, L., and Clow, J. 1996. Quickset: Multimodal Interaction for Distributed Applications. Proceedings of ACM Multimedia, 31-40.
10. Eisenstein J. and Christoudias. C. 2004. A salience-based approach to gesture-speech alignment. In Proceedings of HLT/NAACL'04.
11. Gildea, D. and Hofmann, T. 1999. Topic-based language models using EM. In Proceedings of Eurospeech.
12. Griffin, Z. M. 2001. Gaze durations during speech reflect word selection and phonological encoding. Cognition 82, B1-B14. (Pubitemid 32979187)
13. Grosz, B. J., Joshi, A. K., and Weinstein, S. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2).
14. Grice, H. P. Logic and Conversation. 1975. In Cole, P., and Morgan, J., eds. Speech Acts. New York, New York: Academic Press. 41-58.
15. Gundel, J. K., Hedberg, N., and Zacharski, R. 1993. Cognitive Status and the Form of Referring Expressions in Discourse. Language 69(2):274-307.
16. Harper, M.., White, C., Wang, W., Johnson, M., and Helzerman, R. 2000. The Effectiveness of Corpus-Induced Dependency Grammars for Post-processing Speech. Proceedings of the North American Association for Computational Linguistics, 102-109.
17. Heeman. P. 1999. POS tags and decision trees for language modeling. In Proceedings of the Conference on Empirical Methods in Natural Language Process (EMNLP).
18. Huls, C., Bos, E., and Classen, W. 1995. Automatic Referent Resolution of Deictic and Anaphoric Expressions. Computational Linguistics, 21(1):59-79.
19. Jelinek, F. 1990. Self-organized language modeling for speech recognition. In Waibel, A. and Lee, K. F. (Eds). Readings in Speech Recognition, pp. 450-506.
20. Johnston, M. 1998. Unification-based Multimodal parsing, Proceedings of COLING-ACL.
21. Johnston, M., Bangalore, S., Visireddy G., Stent, A., Ehlen, P., Walker, M., Whittaker, S., and Maloor, P. 2002. MATCH: An Architecture for Multimodal Dialog Systems, in Proceedings of the 40th ACL, Philadelphia, pp. 376-383.
22. Katz, S. M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3).
23. Kehler, A. 2000. Cognitive Status and Form of Reference in Multimodal Human-Computer Interaction, Proceedings of AAAI'01.
24. Kneser, R. and Ney, H. 1993. Improved clustering techniques for class-based statistical language modeling. In Eurospeech'93, pp. 973-976.
25. Landragin, F., Bellalem, N., and Romary, L. 2001. Visual Salience and Perceptual Grouping in Multimodal Interactivity. In: First International Workshop on Information Presentation and Natural Multimodal Dialogue, Verona, Italy, pp. 151-155.
26. Lappin, S., and Leass, H. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4).
27. Oviatt, S. 1999. Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. In Proceedings of CHI.
28. Pieraccini, R., Dayandhi, K., Bloom, J., Dahan, J.-G., Phillips, M., Goodman, B. R., Prasad, K. V., 2004. Multimodal Conversational Systems for Automobiles, Communications of the ACM, Vol. 47, No. 1, pp. 47-49.
29. Roy, D. and Mukherjee, N. 2005. Towards Situated Speech Understanding: Visual Context Priming of Language Models. Computer Speech and Language, 19(2): 227-248.
30. Samuelsson, C. and Reichl, W. 1999. A class-based Language Model for Large Vocabulary Speech Recognition Extracted from Part-of-Speech Statistics. In IEEE ICASSP'99.
31. Schuler, W. 2003. Using model-theoretic semantic interpretation to guide statistical parsing and word recognition in a spoken language interface. In Proceedings of ACL, Sapporo, Japan.
32. Wai, C., Pierraccinni, R., and Meng, H. 2001. A Dynamic Semantic Model for Rescoring Recognition Hypothesis. Proceedings of the ICASSP.
33. Wang, W. and Harper, M. 2002. The superARV language model: In Investigating the effectiveness of tightly integrating multiple knowledge sources. In Proceedings of EMNLP, 238-247.
34. Zue, V., Glass, J., Goodine, D., Leung, H., Phillips, M., Polifroni, J., and Seneff, S. 1991. Integration of Speech Recognition and Natural Language Processing in the MIT Voyager System. Proceedings of the ICASSP.