Of words, eyes and brains: Correlating image-based distributional semantic models with neural representations of concepts

EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference

Volumn , Issue , 2013, Pages 1960-1970

  Anderson, Andrew J ^a   Bruni, Elia ^a   Bordignon, Ulisse ^a   Poesio, Massimo ^a   Baroni, Marco ^a  

^a UNIVERSITY OF TRENTO   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS;

CO-OCCURRENCE PATTERN; IMAGE COLLECTIONS; IMAGE-BASED MODELS; NEURAL REPRESENTATIONS; SEMANTIC PATTERN; SEMANTIC REPRESENTATION; SEMANTIC SIMILARITY; VISUAL PERCEPTION;

SEMANTICS;

EID: 84893352999     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (44)

1. Shane Bergsma and Randy Goebel. 2011. Using visual information to predict lexical preference. In Proceedings of RANLP, pages 399-405, Hissar, Bulgaria.
2. Jeffrey R. Binder, Rutvik H. Desai, William W. Graves, and Lisa L. Conant. 2009. Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortexl, 12:2767-2796.
3. Vicki Bruce, Patrick R Green, and Georgeson Mark A. 2003. Visual perception: Physiology, psychology, and ecology. Psychology Pr.
4. Elia Bruni, Giang Binh Tran, and Marco Baroni. 2011. Distributional semantics from text and images. In Proceedings of the EMNLP GEMS Workshop, pages 22-32, Edinburgh, UK.
5. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. 2012a. Distributional semantics in Technicolor. In Proceedings of ACL, pages 136-145, Jeju Island, Korea.
6. Elia Bruni, Jasper Uijlings, Marco Baroni, and Nicu Sebe. 2012b. Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning. In Proceedings of ACM Multimedia, pages 1219-1228, Nara, Japan.
7. Elia Bruni, Ulisse Bordignon, Adam Liska, Jasper Uijlings, and Irina Sergienya. 2013. Vsem: An open library for visual semantics representation. In Proceedings of ACL, Sofia, Bulgaria.
8. John Bullinaria and Joseph Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39:510-526.
9. Kai-min Chang, Tom Mitchell, and Marcel Just. 2011. Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation. NeuroImage, 56:716-727.
10. Linda L Chao, James V Haxby, and Alex Martin. 1999. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature neuroscience, 2(10):913-919.
11. Kenneth Church and Peter Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29.
12. Stephen Clark. 2012. Vector space models of lexical meaning. In Shalom Lappin and Chris Fox, editors, Handbook of Contemporary Semantics, 2nd edition. Blackwell, Malden, MA. In press.
13. Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. 2004. Visual categorization with bags of keypoints. In In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1-22, Prague, Czech Republic.
14. Jia Deng, Wei Dong, Richard Socher, Lia-Ji Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of CVPR, pages 248-255, Miami Beach, FL.
15. Russell A Epstein. 2008. Parahippocampal and ret-rosplenial contributions to human spatial navigation. Trends in cognitive sciences, 12(10):388-396.
16. Katrin Erk. 2012. Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10):635-653.
17. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA.
18. Yansong Feng and Mirella Lapata. 2010. Visual information in semantic representation. In Proceedings of HLT-NAACL, pages 91-99, Los Angeles, CA.
19. Melvyn A. Goodale and David Milner. 1992. Separate visual pathways for perception and action. Trends in Neurosciences, 15:20-25.
20. Kristen Grauman and Bastian Leibe. 2011. Visual Object Recognition. Morgan & Claypool, San Francisco.
21. Peter Hagoort. 2005. On Broca, brain, and binding: a new framework. Trends in cognitive sciences, 9(9):416-423.
22. James Haxby, Ida Gobbini, Maura Furey, Alumit Ishai, Jennifer Schouten, and Pietro Pietrini. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293:2425-2430.
23. Alexander Huth, Shinji Nishimoto, An Vu, and Jack Gallant. 2012. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6):1210-1224.
24. Nancy Kanwisher and Galit Yovel. 2006. The fusiform face area: a cortical region specialized for the perception of faces. Philosophical Transactions of the Royal Society B: Biological Sciences, 361(1476):2109-2128.
25. Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. 2008. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2.
26. Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. 2006. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proceedings of CVPR, pages 2169-2178, Washington, DC.
27. Chee Wee Leong and Rada Mihalcea. 2011. Going beyond text: A hybrid image-text approach for measuring word relatedness. In Proceedings of IJCNLP, pages 1403-1407.
28. Max Louwerse. 2011. Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, 3:273-302.
29. David G Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60:91-110.
30. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical cooccurrence. Behavior Research Methods, 28:203-208.
31. Bruce D McCandliss, Laurent Cohen, and Stanislas Dehaene. 2003. The visual word form area: expertise for reading in the fusiform gyrus. Trends in cognitive sciences, 7(7):293-299.
32. Robert D McIntosh and Thomas Schenk. 2009. Two visual streams for perception and action: Current trends. Neuropsychologia, 47(6):1391-1396.
33. Earl K Miller, David J Freedman, and Jonathan D Wallis. 2002. The prefrontal cortex: categories, concepts and cognition. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 357(1424):1123-1136.
34. Tom Mitchell, Svetlana Shinkareva, Andrew Carlson, Kai-Min Chang, Vincente Malave, Robert Mason, and Marcel Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science, 320:1191-1195.
35. ∗SEM, pages 114-123, Montreal, Canada.
36. Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom Mitchell. 2009. Zero-shot learning with semantic output codes. In Proceedings of NIPS, pages 1410-1418, Vancouver, Canada.
37. Marius V Peelen and Paul E Downing. 2005. Selectivity for the human body in the fusiform gyrus. Journal of Neurophysiology, 93(1):603-608.
38. Francisco Pereira, Greg Detre, and Matthew Botvinick. 2011. Generating text from functional brain images. Frontiers in Human Neuroscience, 5(72). Published online: http://www.frontiersin.org/human-neuroscience/10.3389/fnhum.2011.00072/abstract.
39. Alexander T Sack. 2009. Parietal cortex and spatial cognition. Behavioural brain research, 202(2):153-161.
40. Hinrich Schütze. 1997. Ambiguity Resolution in Natural Language Learning. CSLI, Stanford, CA.
41. Carina Silberer and Mirella Lapata. 2013. Models of semantic representation with visual attributes. In Proceedings of ACL, Sofia, Bulgaria.
42. Josef Sivic and Andrew Zisserman. 2003. Video Google: A text retrieval approach to object matching in videos. In Proceedings of ICCV, pages 1470-1477, Nice, France.
43. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141-188.
44. N Tzourio-Mazoyer, B Landeau, D Papathanassiou, F Crivello, O Etard, N Delcroix, B Mazoyer, and M Joliot. 2002. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcel-lation of the MNI MRI single-subject brain. Neuroimage, 15(1):273-289.