What's in a Gaze? The role of eye-gaze in reference resolution in multimodal conversational interfaces

International Conference on Intelligent User Interfaces, Proceedings IUI

Volumn , Issue , 2008, Pages 20-29

  Prasov, Zahar ^a   Chai, Joyce Y ^a  

^a Michigan State University   (United States)

Author keywords

Eyc gaze; Multimodal conversational interfaces; Reference resolution

Indexed keywords

SPEECH;

COMPLEX DOMAINS; CONVERSATIONAL INTERFACES; DO-MAINS; DOMAIN MODELS; EMPIRICAL RESULTS; EYC-GAZE; EYE GAZES; GRAPHICAL DISPLAYS; LINGUISTIC PROCESSING; MULTI MODALS; MULTIMODAL CONVERSATIONAL INTERFACES; PROBABILISTIC FRAMEWORKS; REFERENCE RESOLUTION; REFERENCE RESOLUTIONS;

USER INTERFACES;

EID: 57349198814     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1378773.1378777     Document Type: Conference Paper

References (25)

1. D. K. Byron. Resolving pronominal reference to abstract entities. In In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL '02), 2002.
2. D. K. Byron, T. Mampilly, and T. Sharma, V.and Xu. Utilizing visual attention for cross-modal corcfcrence interpretation. In Spring Lecture Notes in Computer Science: Proceedings of CONTEXT-O5, pages 83-96, 2005.
3. E. Campana, J. Baldridge, J. Dowding, B. A. Hockey, R. W. Remington, and L. S. Stone. Using eye movements to determine referents in a spoken dialogue system. In Proceedings of Perceptive User Interfaces, 2001.
4. J. Chai, P. Hong, M. Zhou, and Z Prasov. Optimization in multimodal interpretation. In Proceedings of 42nd Annual Meeting of Association for Computational Linguistics (ACL), pages 1-8, 2004.
5. J. Chai, Z. Prasov, and S. Qu. Cognitive principles in robust multimodal interpretation. Journal of Artificial Intelligence Research, 27:55-83, 2006.
6. J. Y. Chai, Z. Prasov, J. Blaim, and R. Jin. Linguistic theories in efficient multimodal reference resolution: An empirical investigation. In ACM International Conference of Intelligent User Interfaces (IUI05). ACM Press, 2005.
7. J. M. Findlay. Eye scanning and visual search. In J. M. Henderson and F. Ferrcira, editors, The interface of language, vision, and action: Eye movements and the visual world. New York: Psychology Press, 2004.
8. H. P. Gricc. Speech acts. In Logic and conversation, pages 41-58. New York: Academic Press., 1975.
9. Z. M. Griffin. Gaze durations during speech reflect word selection and phonological encoding. In Cognition, volume 82, pages B1-B14, 2001.
10. Z. M. Griffin and K. Bock. What the eyes say about speaking. In Psychological Science, volume 11, pages 274-279, 2000.
11. J.M. Henderson and F. Ferrcira. In The interface of language, vision, and action: Eye movements and the visual world. Taylor & Francis, 2004.
12. R. J. K. Jacob. Eye tracking in advanced interface design. In W. Barfield and T. Furness, editors, Advanced Interface Design and Virtual Environments, pages 258-288, 1995.
13. M. Johnston. Unification-based multimodal parsing. In Proceedings of ACL/COLING'98, 1998.
14. M. Johnston and S Bangalore. Finite-state multimodal parsing and understanding. In In Proceedings of COLING00, 2000.
15. M. A. Just and P. A. Carpenter. Eye fixations and cognitive processes. In Cognitive Psychology, volume 8, pages 441-480, 1976.
16. M. Kaur, M. Tremainc, N. Huang, J. Wilder, Z. Gacovski, F. Flippo, and C. S. Mantravadi. Where is "it"? event synchronization in gaze-speech input systems. In Proceedings of Fifth International Conference on Multimodal Interfaces, pages 151-157. ACM Press, 2003.
17. A. Kehlcr. Cognitive status and form of reference in multimodal human-computer interaction. In In Proceedings of AAAI00, 2000.
18. J. Kelleher and J. van Genabith. Visual salience and reference resolution in simulated 3-d environments. Artificial Intelligence Review, 21(3), 2004.
19. E. Matin. Saccadic suppression: a review and an analysis. In Psychological Bulletin, volume 81, pages 899-917, 1974.
20. A. S. Meyer and W. J. M. Levclt. Viewing and naming objects: Eye movements during noun phrase production. In Cognition, volume 66, pages B25-B33, 1998.
21. Z. Prasov, J. Chai, and H. Jeong. Eye gaze for attention prediction in multimodal human-machine conversation. Technical report, In Proceedings of the AAAI Spring Symposium on Interaction Challenges for Intelligent Assistants, 2007.
22. S. Qu and J. Chai. An exploration of eye gaze in spoken language processing for multimodal conversational interfaces. In In Proceedings of the Conference of the North America Chapter of the Association of Computational Linguistics (NAACL), 2007.
23. P. Qvarfordt, D. Beymer, and S. Zhai. Realtourist - a study of augmenting human-human and human-computer dialogue with eye-gaze overlay. In INTERACT 2005, LNCS 3585, pages 767-780, 2005.
24. P. Qvarfordt and S. Zhai. Conversing with the user based on eye-gaze patterns. In Proc. Of the Conference on Human Factors in Computing Systems. ACM, 2005.
25. M. K. Tanenhous, M. Spivey-Knowlton, E. Eberhard, and J. Sedivy. Integration of visual and linguistic information during spoken language comprehension. In Science, volume 268, pages 1632-1634, 1995.