Context-based word acquisition for situated dialogue in a virtual world

Journal of Artificial Intelligence Research

Volumn 37, Issue , 2010, Pages 247-277

  Qu, Shaolin ^a   Chai, Joyce Y ^a  

^a Michigan State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACQUISITION PROCESS; AUTOMATED RECOGNITION; CONTEXT-BASED; CONTEXTUAL INFORMATION; CONVERSATIONAL SYSTEMS; DOMAIN KNOWLEDGE; EYE-GAZE; HUMAN EYE; HUMAN LANGUAGE; HUMAN-MACHINE; KNOWLEDGE BASE; LANGUAGE PRODUCTION; NEW APPROACHES; VIRTUAL WORLDS; WORD ACQUISITION;

EYE MOVEMENTS; KNOWLEDGE BASED SYSTEMS; LINGUISTICS; SPEECH RECOGNITION; VIRTUAL REALITY;

MERGERS AND ACQUISITIONS;

EID: 77952680332     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.2912     Document Type: Article

References (54)

1. Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory & Language, 38, 419-439.
2. Barnard, K., Duygulu, P., de Freitas, N., Forsyth, D., Blei, D., & Jordan, M. (2003). Match- ing words and pictures. Journal of Machine Learning Research, 3, 1107-1135.
3. Bock, K., Irwin, D. E., Davidson, D. J., & Leveltb, W. (2003). Minding the clock. Journal of Memory and Language, 48, 653-685.
4. Brown, P. F., Pietra, S. D., Pietra, V. J. D., & Mercer, R. L. (1993). The mathematic of statistical machine translation: Parameter estimation. Computational Linguistics, 19 (2), 263-311.
5. Brown-Schmidt, S., & Tanenhaus, M. K. (2006). Watching the eyes when talking about size: An investigation of message formulation and utterance planning. Journal of Memory and Language, 54, 592-609.
6. Byron, D., Mampilly, T., Sharma, V., & Xu, T. (2005). Utilizing visual attention for cross-modal coreference interpretation. In Proceedings of the Fifth International and Interdisciplinary Conference on Modeling and Using Context (CONTEXT-05), pp. 83-96.
7. Campana, E., Baldridge, J., Dowding, J., Hockey, B., Remington, R., & Stone, L. (2001). Using eye movements to determine referents in a spoken dialogue system. In Proceed- ings of the Workshop on Perceptive User Interface.
8. Cessie, S. L., & Houwelingen, J. V. (1992). Ridge estimators in logistic regression. Applied Statistics, 41 (1), 191-201.
9. Chen, D. L., & Mooney, R. J. (2008). Learning to sportscast: A test of grounded language acquisition. In Proceedings of 25th International Conference on Machine Learning (ICML).
10. Cooke, N. J. (2006). Gaze-Contigent Automatic Speech Recognition. Ph.D. thesis, University of Birminham.
11. Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin & Review, 12 (3), 453-459.
12. DeVault, D., & Stone, M. (2003). Domain inference in incremental interpretation. In Proceedings of ICoS.
13. Eberhard, K., Spivey-Knowiton, M., Sedivy, J., & Tanenhaus, M. (1995). Eye movements as a window into real-time spoken language comprehension in natural contexts. Journal of Psycholinguistic Research, 24, 409-436.
14. Fang, R., Chai, J. Y., & Ferreira, F. (2009). Between linguistic attention and gaze fixations inmultimodal conversational interfaces. In Proceedings of the International Conference on Multimodal Interfaces (ICMI), pp. 143-150.
15. Fazly, A., Alishahi, A., & Stevenson, S. (2008). A probabilistic incremental model of word learning in the presence of referential uncertainty. In Proceedings of the 30th Annual Conference of the Cognitive Science Society.
16. Fellbaum, C. (Ed.). (1998). WordNet: An Electronic Lexical Database. MIT Press.
17. Fleischman, M., & Roy, D. (2005). Intentional context in situated language learning. In Proceedings of the 9th Conference on Computational Natural Language Learning (CoNLL).
18. Fong, T. W., & Nourbakhsh, I. (2005). Interaction challenges in human-robot space explo- ration. Interactions, 12 (2), 42-45.
19. Gorniak, P., & Roy, D. (2004). Grounded semantic composition for visual scenes. Journal of Artificial Intelligence Research, 21, 429-470.
20. Grifin, Z., & Bock, K. (2000). What the eyes say about speaking. Psychological Science, 11, 274-279.
21. Grifin, Z. M. (2001). Gaze durations during speech reect word selection and phonological encoding. Cognition, 82, B1-B14.
22. Grifin, Z. M. (2004). Why look? Reasons for eye movements related to language production. In Henderson, J., & Ferreira, F. (Eds.), The Interface of Language, Vision, and Action: Eye Movements and the Visual World, pp. 213-248. Taylor and Francis.
23. Grosz, B. J., Joshi, A. K., & Weinstein, S. (1995). Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21 (2), 203-226.
24. Jacob, R. J. K. (1991). The use of eye movements in human-computer interaction techniques: What you look at is what you get. ACM Transactions on Information Systems, 9 (3), 152-169.
25. Kahneman, D. (1973). Attention and Effort. Prentice-Hall, Inc., Englewood Cliffs.
26. Kaur, M., Termaine, M., Huang, N., Wilder, J., Gacovski, Z., Flippo, F., & Mantravadi, C. S. (2003). Where is "it"? event synchronization in gaze-speech input systems. In Proceedings of the International Conference on Multimodal Interfaces (ICMI).
27. Klemmer, S., Sinha, A., Chen, J., Landay, J., Aboobaker, N., & Wang, A. (2000). SUEDE: A wizard of oz prototyping tool for speech user interfaces. In Proceedings of ACM Symposium on User Interface Software and Technology, pp. 1-10.
28. Lemon, O., Gruenstein, A., & Peters, S. (2002). Collaborative activities and multitasking in dialogue systems. Traitement Automatique des Langues, 43 (2), 131-154.
29. Liang, P., Jordan, M. I., & Klein, D. (2009). Learning semantic correspondences with less supervision. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics (ACL).
30. Liu, Y., Chai, J., & Jin, R. (2007). Automated vocabulary acquisition and interpretation in multimodal conversational systems. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL).
31. Meyer, A., Sleiderink, A., & Levelt, W. (1998). Viewing and naming objects: eye movements during noun phrase production. Cognition, 66 (22), 25-33.
32. Nakano, Y., Reinstein, G., Stocky, T., & Cassell, J. (2003). Towards a model of face-to-face grounding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
33. Pedersen, T., Patwardhan, S., & Michelizzi, J. (2004). WordNet::Similarity - measuring the relatedness of concepts. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI).
34. Prasov, Z., & Chai, J. Y. (2008). What's in a gaze? the role of eye-gaze in reference resolution in multimodal conversational interfaces. In Proceedings of ACM 12th International Conference on Intelligent User interfaces (IUI).
35. Qu, S., & Chai, J. Y. (2007). An exploration of eye gaze in spoken language processing for multimodal conversational interfaces. In Proceedings of the Human Language Technol- ogy Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pp. 284-291.
36. Qu, S., & Chai, J. Y. (2008). Incorporating temporal and semantic information with eye gaze for automatic word acquisition in multimodal conversational systems. In Proceedingsof the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 244-253.
37. Qu, S., & Chai, J. Y. (2009). The role of interactivity in human-machine conversation for automatic word acquisition. In Proceedings of the 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 188-195.
38. Qvarfordt, P., & Zhai, S. (2005). Conversing with the user based on eye-gaze patterns. In Proceedings of the Conference on Human Factors in Computing Systems (CHI).
39. Rayner, K. (1998). Eye movements in reading and information processing - 20 years of research. Psychological Bulletin, 124 (3), 372-422.
40. Roy, D. (2002). Learning visually-grounded words and syntax for a scene description task. Computer Speech and Language, 16 (3), 353-385.
41. Roy, D., & Pentland, A. (2002). Learning words from sights and sounds, a computational model. Cognitive Science, 26 (1), 113-146.
42. Schuler, W., Wu, S., & Schwartz, L. (2009). A framework for fast incremental interpretation during speech decoding. Computational Linguistics, 35 (3), 313-343.
43. Siskind, J. M. (2001). Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. Journal of Artificial Intelligence Research, 15, 31-90.
44. Spivey, M. J., Tanenhaus, M. K., Eberhard, K. M., & Sedivy, J. C. (2002). Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481.
45. Taguchi, R., Iwahashi, N., Nose, T., Funakoshi, K., & Nakano, M. (2009). Learning lexi- cons from spoken utterances based on statistical model selection. In Proceedings of Interspeech.
46. Tanaka, K. (1999). A robust selection system using real-time multi-modal user-agent inter- actions. In Proceedings of the International Conference on Intelligent User Interfaces (IUI).
47. Tanenhaus, M., Spivey-Knowiton, M., Eberhard, K., & Sedivy, J. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.
48. Toutanova, K., Klein, D., Manning, C., & Singer, Y. (2003). Feature-Rich Part-of-Speech tagging with a cyclic dependency network. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Com- putational Linguistics (HLT-NAACL), pp. 252-259.
49. Toutanova, K., & Manning, C. D. (2000). Enriching the knowledge sources used in a max- imum entropy part-of-speech tagger. In Proceedings of the Joint SIGDAT Confer- ence on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC), pp. 63-70.
50. Traum, D., & Rickel, J. (2002). Embodied agents for multiparty dialogue in immersive virtual worlds. In Proceedings of the 1st international joint conference on Autonomous Agents and Multi-Agent Systems.
51. Wang, J. (1995). Integration of eye-gaze, voice and manual response in multimodal user interfaces. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, pp. 3938-3942.
52. Yu, C., & Ballard, D. (2004). A multimodal learning interface for grounding spoken language in sensory perceptions. ACM Transactions on Applied Perceptions, 1 (1), 57-80.
53. Zhai, S., Morimoto, C., & Ihde, S. (1999). Manual and gaze input cascaded (MAGIC) pointing. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pp. 246-253.
54. Zhang, Q., Imamiya, A., Go, K., & Mao, X. (2004). Overriding errors in a speech and gaze multimodal architecture. In Proceedings of the International Conference on Intelligent User Interfaces (IUI).