Between linguistic attention and gaze fixations inmultimodal conversational interfaces

ICMI-MLMI'09 - Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interfaces

Volumn , Issue , 2009, Pages 143-150

  Fang, Rui ^a   Chai, Joyce Y ^a   Ferreira, Fernanda ^b  

^a Michigan State University   (United States)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Gaze fixations; Linguistic attention; Multimodal conversational interfaces

Indexed keywords

CONVERSATIONAL INTERFACE; EMPIRICAL FINDINGS; HUMAN ATTENTION; HUMAN-MACHINE; LINGUISTIC PROCESSING; MULTI-MODAL; MULTIMODAL CONVERSATIONAL INTERFACES; VISUAL ATTENTION; VISUAL-PROCESSING; WORD ACQUISITION;

INTERACTIVE COMPUTER SYSTEMS; LEARNING SYSTEMS; VISUAL COMMUNICATION;

LINGUISTICS;

EID: 74049119292     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1647314.1647339     Document Type: Conference Paper

References (26)

1. K. Bock, D. Irwin, and D. Davidson. Putting first things first. In J. M. Henderson and F. Ferreira, editors, The Interface of Language, Vision, and Action: Eye Movements and the visual world, pages 249-278. New York: Psychology Press, 2004.
2. S. E. Brennan, M. W. Friedman, and C. Pollard. A centering approach to pronouns. In In Proceedings of 25th Annual Meeting of Association for Computational Linguistics, pages 155-162, 1987.
3. D. Byron, T. Mampilly, V. Sharma, and T. Xu. Utilizing visual attention for cross-modal coreference interpretation. In Proceedings of the Fifth International and Interdisciplinary Conference on Modeling and Using Context (CONTEXT-05), pages 83-96, 2005.
4. J. Chai, P. Hong, M. Zhou, and Z. Prasov. Optimization in multimodal interpretation. In Proceedings of 42nd Annual Meeting of Association for Computational Linguistics (ACL), pages 1-8, 2004.
5. P. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. Quickset: Multimodal interaction for distributed applications. In Proceedings of ACM Multimedia, page 311C 40, 1996.
6. N. J. Cooke and M. Russell. Gaze-contingent asr for spontaneous, conversational speech: an evaluation. In International Conference in Acoustics, Speech and Signal Processing)., 2008.
7. Z. M. Griffin. Gaze durations during speech reflect word selection and phonological encoding. Cognition, 82:B1-B14, 2001.
8. Z. M. Griffin. Why look? reasons for eye movements related to language production. The Interface of Language, Vision, and Action, pages 213-247, 2004.
9. B. J. Grosz, A. K. Joshi, and S. Weinstein. centering: a framework for modeling the local coherence of discourse. Computational Linguistics, 21:203-225, 1995.
10. B. J. Grosz and C. Sidner. Attention, intention, and the structure of discourse. Computational Linguistics, 12(3):175-204, 1986.
11. J. Henderson and F. Ferreira. The Interface of Language, Vision, and Action: Eye Movements and Visual World. Taylor and Francis, New York, 2004.
12. L. Itti and C. Koch. Computational modelling of visual attention. Nat Rev Neurosci, 2:194-203, 2001.
13. M. Johnston, S. Bangalore, G. Visireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, and P. Maloor. Match: An architecture for multimodal dialog systems. In in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 376-383, 2002.
14. M. Just and P. Carpenter. Eye fixations and cognitive processes. Cognitive Psychology, 8:441-480, 1976.
15. S. Kullback. The kullback-leibler distance. The American Statistician, 41:340-341, 1987.
16. J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37:145-151, 1991.
17. Y. Liu, J. Y. Chai, and R. Jin. Automated vocabulary acquisition and interpretation in multimodal conversational systems. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL), 2007.
18. A. S. Meyer, A. Sleiderink, and W. J. M. Levelt. Viewing and naming objects: Eye movements during noun phrase production. Cognition, 66:B25-B33, 1998.
19. S. L. Oviatt. Mutual disambiguation of recognition errors in a multimodal architecture. In In Proceedings of Conference on Human Factors in Computing Systems: CHI '99, 1999.
20. Z. Prasov and J. Y. Chai. What's in a gaze? the role of eye-gaze in reference resolution in multimodal conversational interfaces. In Proceedings of ACM 12th International Conference on Intelligent User interfaces (IUI), pages 20-29, January 2008.
21. Z. Prasov, J. Y. Chai, and H. Jeong. Eye gaze in attention prediction in multimodal human machine conversation. In Proceedings of the AAAI 2007 Spring Symposium on Interaction Challenges for Artificial Assistants, March 2007.
22. S. Qu and J. Y. Chai. An exploration of eye gaze in spoken language processing for multimodal conversational interfaces. In Proceedings of the Conference of the North America Chapter of the Association of Computational Linguistics (NAACL), pages 284-291, April 2007.
23. S. Qu and J. Y. Chai. Incorporating temporal and semantic information with eye gaze for automatic word acquisition in multimodal conversational systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2008.
24. F. Quek, D. McNeill, R. Bryll, S. Duncan, X. Ma, C. Kirbas, K. McCullough, and R. Ansari. Multimodal human discourse gesture and speech. ACM Transactions on Computer-Human Interaction, 9:171-193, 2002.
25. R. Sharma, S. Kettevekov, and M. Yeasin. Integration of Gesture and Speech in Multimodal Interface. Computer Science Handbook for Displays, 2001.
26. M. K. Tenenhaus, M. Sivey-Knowlton, E. Eberhard, and J. Sedivy. Integration of visual and linguistic information during spoken language comprehension. Science, 268:1632-1634, 1995.