Do you see what I mean? Visual resolution of linguistic ambiguities

Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing

Volumn , Issue , 2015, Pages 1477-1487

  Berzak, Yevgeni ^a   Barbu, Andrei ^a   Harari, Daniel ^a   Katz, Boris ^a   Ullman, Shimon ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b WEIZMANN INSTITUTE OF SCIENCE   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS; SPECTRAL RESOLUTION;

CONTEXTUAL INFORMATION; HAND IN HANDS; LANGUAGE UNDERSTANDING; MULTI-MODAL; NOVEL TASK; VISION MODEL; VISUAL RESOLUTIONS; VISUAL SCENE;

VISUAL LANGUAGES;

EID: 84959867661     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.18653/v1/d15-1172     Document Type: Conference Paper

References (31)

1. Kobus Barnard, Matthew Johnson, and David Forsyth. 2003. Word sense disambiguation with pictures. In Proceedings of the HLT-NAACL 2003 workshop on Learning word meaning from non-linguistic data-Volume 6, pages 1-5. Association for Computational Linguistics.
2. L. E. Baum, T. Petrie, G. Soules, and N. Weiss. 1970. A maximization technique occuring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics, 41(1):164-171.
3. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics.
4. Moreno I Coco and Frank Keller. 2015. The interaction of visual and linguistic saliency during syntactic ambiguity resolution. The Quarterly Journal of Experimental Psychology, 68(1):46-74.
5. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In Computer Vision-ECCV 2010, pages 15-29. Springer.
6. Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. 2010. Object detection with discriminatively trained part-based models. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(9):1627-1645.
7. Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hockenmaier, and Svetlana Lazebnik. 2014. Improving image-sentence embeddings using large weakly annotated photo collections. In Computer Vision-ECCV 2014, pages 529-545. Springer.
8. Andrej Karpathy and Li Fei-Fei. 2014. Deep visualsemantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306.
9. Evan Kidd and Judith Holler. 2009. Children's use of gesture to resolve lexical ambiguity. Developmental Science, 12(6):903-913.
10. Chen Kong, Dahua Lin, Mayank Bansal, Raquel Urtasun, and Sanja Fidler. 2014. What are you talking about? text-to-image coreference. In Computer Vision and Pattern Recognition (CVPR), pages 3558-3565. IEEE.
11. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105.
12. Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. 2011. Hmdb: a large video database for human motion recognition. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2556-2563. IEEE.
13. G. Kulkarni, V. Premraj, S. Dhar, Siming Li, Yejin Choi, AC Berg, and TL Berg. 2011. Baby talk: Understanding and generating simple image descriptions. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, pages 1601-1608. IEEE Computer Society.
14. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. CoRR, abs/1501.02598.
15. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014, pages 740-755. Springer.
16. Jingen Liu, Jiebo Luo, and Mubarak Shah. 2009. Recognizing realistic actions from videos in the wild. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 1996-2003. IEEE.
17. Margaret Mitchell, Xufeng Han, Jesse Dodge, Alyssa Mensch, Amit Goyal, Alex Berg, Kota Yamaguchi, Tamara Berg, Karl Stratos, and Hal Daumé III. 2012. Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapter of the Associationfor Computational Linguistics, pages 747-756. Association for Computational Linguistics.
18. Vignesh Ramanathan, Armand Joulin, Percy Liang, and Li Fei-Fei. 2014. Linking people in videos with their names using coreference resolution. In Computer Vision-ECCV 2014, pages 95-110. Springer.
19. Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36.
20. Mikel D. Rodriguez, Javed Ahmed, and Mubarak Shah. 2008. Action MACH A Spatio-temporal Maximum Average Correlation Height Filter for Action Recognition. In Computer Vision and Pattern Recognition, pages 1-8.
21. Narayanaswamy Siddharth, Andrei Barbu, and Jeffrey Mark Siskind. 2014. Seeing what you're told: Sentence-guided activity recognition in video. In Computer Vision and Pattern Recognition (CVPR), pages 732-739. IEEE.
22. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, pages 721-732.
23. Catherine E Snow. 1972. Mothers' speech to children learning language. Child development, pages 549-565.
24. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems, pages 935-943.
25. Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207-218.
26. Michael J Spivey, Michael K Tanenhaus, Kathleen M Eberhard, and Julie C Sedivy. 2002. Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive psychology, 45(4):447-481.
27. Michael K Tanenhaus, Michael J Spivey-Knowlton, Kathleen M Eberhard, and Julie C Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217):1632-1634.
28. Jesse Thomason, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Raymond Mooney. 2014. Integrating language and vision to generate natural language descriptions of videos in the wild. In Proceedings of the 25th International Conference on Computational Linguistics (COLING), August.
29. Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. 2015. Translating videos to natural language using deep recurrent neural networks. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado.
30. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition (CVPR).
31. A. J. Viterbi. 1971. Convolutional codes and their performance in communication systems. Communications of the IEEE, 19:751-772, October.