Scan Patterns Predict Sentence Production in the Cross-Modal Processing of Visual Scenes

Cognitive Science

Volumn 36, Issue 7, 2012, Pages 1204-1223

  Coco, Moreno I ^a   Keller, Frank ^a  

^a UNIVERSITY OF EDINBURGH   (United Kingdom)

Author keywords

Cross model processing; Eye movements; Language production; Scan patterns; Scene understanding; Similarity measures

Indexed keywords

ALGORITHM; ARTICLE; ATTENTION; BIOLOGY; EYE MOVEMENT; HUMAN; LINGUISTICS; MENTAL FUNCTION; METHODOLOGY; PATTERN RECOGNITION; PERCEPTIVE DISCRIMINATION; PHOTOSTIMULATION; PSYCHOLOGICAL MODEL; SPEECH ANALYSIS; VERBAL BEHAVIOR;

ALGORITHMS; ATTENTION; COMPUTATIONAL BIOLOGY; EYE MOVEMENTS; HUMANS; MENTAL PROCESSES; MODELS, PSYCHOLOGICAL; PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; PSYCHOLINGUISTICS; SIGNAL DETECTION, PSYCHOLOGICAL; SPEECH PRODUCTION MEASUREMENT; VERBAL BEHAVIOR;

EID: 84865851108     PISSN: 03640213     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1551-6709.2012.01246.x     Document Type: Article

References (31)

1. Andersson, R., Ferreira, F., & Henderson, J. M. (2011). The integration of complex speech and scenes during language comprehension. Acta Psychologica, 137, 208-216.
2. Baayen, R., Davidson, D., & Bates, D. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412.
3. Castelhano, M., & Heaven, C. (2010). The relative contribution of scene context and target features to visual search in real-world scenes. Attention, Perception and Psychophysics, 72, 1283-1297.
4. Coco, M. I., & Keller, F. (2010). Sentence production in naturalistic scene with referential ambiguity. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 1070-1075). Portland, OR: Cognitive Science Society.
5. Cristino, F., Mathot, S., Theeuwes, J., & Gilchrist, I. (2010). Scanmatch: A novel method for comparing fixation sequences. Behaviour Research Methods, 42, 692-700.
6. Durbin, R., Eddy, S., Krogh, A., & Mitchison, G. (2003). Biological sequence analysis: Probabilistic models of proteins and nucleid acids. Cambridge, UK: Cambridge University Press.
7. Findlay, J. M., & Gilchrist, I. D. (2001). Visual attention: The active vision perspective. In M. Jenkins & L. Harris (Eds.), Vision and attention (pp. 83-103). New York: Springer-Verlag.
8. Foulsham, T., & Underwood, G. (2008). What can saliency models predict about eye movements? Spatial and sequential aspect of fixations during encoding and recognition. Journal of Vision, 8(2), 1-17.
9. Frank, M. C., Goodman, N. D., & Tenenbaum, J. B. (2009). Using speakers referential intentions to model early cross-situational word learning. Psychological Science, 20(5), 578-585.
10. Gleitman, L., January, D., Nappa, R., & Trueswell, J. (2007). On the give and take between event apprehension and utterance formulation. Journal of Memory and Language, 57, 544-569.
11. Griffin, Z., & Bock, K. (2000). What the eyes say about speaking. Psychological science, 11, 274-279.
12. Gusfield, D. (1997). Algorithms on strings, trees and sequences: Computer science and computational biology. Cambridge, UK: Cambridge University Press.
13. Hayhoe, M. (2000). Vision using routines: A functional account of vision. Visual Cognition, 7, 43-64.
14. Henderson, J. M. (2007). Regarding scenes. Current Directions in Psychological Science, 16(4), 219-222.
15. Humphrey, K., & Underwood, G. (2008). Fixation sequences in imagery and in recognition during the processing of pictures of real-world scenes. Journal of Eye Movement Research, 2(2), 1-15.
16. Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40(1), 1489-1506.
17. Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance and world knowledge. Cognitive Science, 30, 481-529.
18. Land, M. F. (2006). Eye movements and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296-324.
19. Land, M. F., Mennie, N., & Rusted, J. (1999). The roles of vision and eye movements in the control of activities of daily living. Perception, 28, 1311-1328.
20. Malcolm, G. L., & Henderson, J. (2010). Combining top-down processes to guide eye movements during real-world scene search. Journal of Vision, 10(2) 1-11.
21. Neider, M.B., Zelinsky, G. (2006). Scene context guides eye movements during visual search. Vision Research, 46, 614-621.
22. Potter, M. (1976). Short-term conceptual memory for pictures. Journal of Experimental Psychology: Human Learning and Memory, 2, 509-522.
23. Prasov, Z., & Chai, J. Y. (2010). Fusing eye gaze with speech recognition hypotheses to resolve exophoric references in situated dialogue. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (pp. 471-481). Cambridge, MA: Association for Computational Linguistics.
24. Qu, S., & Chai, J. (2007). An exploration of eye gaze in spoken language processing for multimodal conversational interfaces. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (pp. 284-291). Rochester: Association for Computational Linguistics.
25. Russell, B., Torralba, A., Murphy, K., & Freeman, W. (2008). Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1-3), 151-173.
26. Schmidt, J., & Zelinsky, G. (2009). Search guidance is proportional to the categorical specificity of a target cue. Quarterly Journal of Experimental Psychology, 62(10), 1904-1914.
27. Tanenhaus, M., Spivey-Knowlton, M., Eberhard, K., & Sedivy, J. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632-634.
28. Torralba, A., Oliva, A., Castelhano, M., & Henderson, J. (2006). Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 4(113), 766-786.
29. Vo, M., & Henderson, J. (2010). The time course of initial scene processing for eye movement guidance in natural scene search. Journal of Vision, 10(3), 1-13.
30. Yang, H., & Zelinsky, G. (2009). Visual search is guided to categorically-defined targets. Vision Research, 49, 2095-2103.
31. Yu, C., & Ballard, D. (2007). A unified model of word learning: Integrating statistical and social cues. Neurocomputing, 70, 2149-2165.