Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text

NeuroImage

Volumn 120, Issue , 2015, Pages 309-322

  Anderson, Andrew James ^a,b   Bruni, Elia ^b   Lopopolo, Alessandro ^b   Poesio, Massimo ^b,c   Baroni, Marco ^b  

^a UNIVERSITY OF ROCHESTER   (United States)

^b Center for Mind/Brain Sciences ^*  (Italy)

^c UNIVERSITY OF ESSEX   (United Kingdom)

Author keywords

Concept representation; Embodiment; Language; Mental imagery; Multimodal semantic models; Perceptual simulation; Representational similarity

Indexed keywords

ARTICLE; CONTROLLED STUDY; FRONTAL LOBE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MATHEMATICAL MODEL; MEASUREMENT ACCURACY; NORMAL HUMAN; OCCIPITAL LOBE; PARIETAL LOBE; PREDICTION; PRIORITY JOURNAL; READING; SEMANTICS; STIMULUS RESPONSE; TEMPORAL LOBE; VISUAL MEMORY; VISUAL STIMULATION; BRAIN CORTEX; BRAIN MAPPING; CONCEPT FORMATION; FEMALE; IMAGE PROCESSING; IMAGINATION; MALE; NUCLEAR MAGNETIC RESONANCE IMAGING; PATTERN RECOGNITION; PHYSIOLOGY; PROCEDURES; THEORETICAL MODEL;

BRAIN MAPPING; CEREBRAL CORTEX; CONCEPT FORMATION; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; IMAGINATION; MAGNETIC RESONANCE IMAGING; MALE; MODELS, THEORETICAL; PATTERN RECOGNITION, VISUAL; READING; SEMANTICS;

EID: 84939450021     PISSN: 10538119     EISSN: 10959572     Source Type: Journal    
DOI: 10.1016/j.neuroimage.2015.06.093     Document Type: Article

References (76)

1. Anderson A.J., Bruni E., Bordignon U., Poesio M., Baroni M. Of words, eyes and brains: Correlating image-based distributional semantic models with neural representations of concepts. Proceedings of EMNLP 2013 1960-1970 2013, Association for Computational Linguistics, Seattle, WA.
2. Andrews M., Vigliocco G., Vinson D. Integrating experiential and distributional data to learn semantic representations. Psychol. Rev. 2009, 116(3):463-498.
3. Barsalou L.W. Simulation, situated conceptualization, and prediction. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2009, 364:1281-1289.
4. Barsalou L.W., Santos A., Simmons W.K., Wilson C.D. Language and simulation in conceptual processing. Symbols, embodiment, and meaning 2008, 245-283. Oxford University Press, Oxford. M. De Vega, A.M. Glenberg, A.C. Graesser (Eds.).
5. Bedny M., Caramazza A. Perception, action, and word meanings in the human brain: the case from action verbs. Ann. N. Y. Acad. Sci. 2011, 1224:81-95.
6. Binder J.R., Desai R.H. The neurobiology of semantic memory. Trends Cogn. Sci. 2011, 15:527-536.
7. Binder J.R., Desai R.H., Graves W.W., Conant L.L. Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cereb. Cortex 2009, 19:2767-2796.
8. Bruffaerts R., Dupont P., Peeters R., De Deyne S., Storms G., Vandenberghe R. Similarity of fMRI activity patterns in left perirhinal cortex reflects similarity between words. J. Neurosci. 2013, 33(47):18597-18607.
9. Bruni E., Bordignon U., Liska A., Uijlings J., Sergienya I. Vsem: An open library for visual semantics representation. Proceedings of ACL 2013, 187-192. (Sofia, Bulgaria).
10. Bruni E., Tran N., Baroni M. Multimodal distributional semantics. J. Artif. Intell. Res. 2014, 49:1-47.
11. Carlson T.A., Simmons R.A., Kriegeskorte N., Slevc L.R. The emergence of semantic meaning in the ventral temporal pathway. J. Cogn. Neurosci. 2014, 26(1):120-131.
12. Chatfield K., Lempitsky V., Vedaldi A., Zisserman A. The devil is in the details: an evaluation of recent feature encoding methods. Proceedings of BMVC 2011.
13. Church K., Hanks P. Word association norms, mutual information, and lexicography. Comput. Linguist. 1990, 16(1):22-29. (DC).
14. Clark S. Vector space models of lexical meaning. Handbook of contemporary semantics 2013, Blackwell, Malden, MA. 2nd ed. S. Lappin, C. Fox (Eds.).
15. Connell L., Lynott D. Principles of representation: why you can't represent the same concept twice. Top. Cogn. Sci. 2014, 6(3):390-406.
16. Connolly A.C., Guntupalli J.S., Gors J., Hanke M., Halchenko Y.O., Wu Y.C., et al. The representation of biological classes in the human brain. J. Neurosci. 2012, 32:2608-2618.
17. Csurka G., Dance C., Fan L., Willamowski J., Bray C. Visual categorization with bags of keypoints. Workshop on statistical learning in computer vision 2004, 1-22. ECCV.
18. Deng J., Dong W., Socher R., Li L., Fei-Fei L. Imagenet: A large-scale hierarchical image database. Proceedings of CVPR 2009, 248-255. IEEE, Miami Beach, FL.
19. Devereux B., Kelly C., Korhonen A. Using fMRI activation to conceptual stimuli to evaluate methods for extracting conceptual representations from corpora. Proceedings of the NAACL HLT 2010 First Workshop on Computational Neurolinguistics 2010, 70-78. Association for Computational Linguistics, Los Angeles, USA. B. Murphy, K.K. Chang, A. Korhonen (Eds.).
20. Devereux B.J., Clarke A., Marouchos A., Tyler L.K. Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects. J. Neurosci. 2013, 33(48):18906-18916.
21. Drucker D.M., Aguirre G.K. Different spatial scales of shape similarity representation in lateral and ventral LOC. Cereb. Cortex 2009, 19(10):2269-2280.
22. Evert, S., 2005. The Statistics of Word Cooccurrences. Ph.D dissertation, Stuttgart University.
23. Fairhall S., Caramazza A. Brain regions that represent amodal conceptual knowledge. J. Neurosci. 2013, 33:10552-10558.
24. Genovese C.R., Lazar N.A., Nichols T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. NeuroImage 2002, 15(4):870-878.
25. Glaser W.R. Picture naming. Cognition 1992, 42:61-105.
26. Grauman K., Darrell T. The pyramid match kernel: Discriminative classification with sets of image features. Proceedings of ICCV 2005, 1458-1465. IEEE, Beijing, China.
27. Grauman K., Leibe B. Visual object recognition 2011, Morgan & Claypool, San Francisco.
28. Hassabis D., Kumaran D., Maguire E.A. Using imagination to understand the neural basis of episodic memory. J. Neurosci. 2007, 27(52):14365-14374.
29. Hauk O., Johnsrude I., Pulvermüller F. Somatotopic representation of action words in human motor and premotor cortex. Neuron 2004, 41(2):301-307.
30. Haxby J.V., Gobbini M.I., Furey M.L., Ishai A., Schouten J.L., Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 2001, 293:2425-2430.
31. Hiramatsu C., Goda N., Komatsu H. Transformation from image-based to perceptual representation of materials along the human ventral visual pathway. NeuroImage 2011, 57(2):482-494.
32. Ishai A., Ungerleider L.G., Haxby J.V. Distributed neural systems for the generation of visual images. Neuron 2000, 28(3):979-990.
33. Johns B., Jones M. Perceptual inference through global lexical similarity. Top. Cogn. Sci. 2012, 4(1):103-120.
34. Just M.A., Cherkassky V.L., Aryal S., Mitchell T.M. A neurosemantic theory of concrete noun representation based on the underlying brain codes. PLoS ONE 2010, 5(1):e8622.
35. Kellenbach M.L., Wijers A.A., Mulder G. Visual semantic features are activated during the processing of concrete words: Event-related potential evidence for perceptual semantic priming. Cogn. Brain Res. 2000, 10(1):67-75.
36. Khaligh-Razavi S.-M., Kriegeskorte N. Deep supervised, but not unsupervised models may explain IT cortical representation. PLoS Comput. Biol. 2014, 10(11):e1003915. 10.1371/journal.pcbi.1003915.
37. Kiefer M., Sim E.-J., Herrnberger B., Grother J., Hoenig K. The sound of concepts: Four markers for a link between auditory and conceptual brain systems. J. Neurosci. 2008, 28(47):12224-12230.
38. Kosslyn S.M., Thompson W.L. When is early visual cortex activated during visual mental imagery?. Psychol. Bull. 2003, 129:723-746.
39. Krause B.J., Schmidt D., Mottaghy F.M., Taylor J., Halsband U., Herzog H., Tellmann L., Müller-Gärtner H.-W. Episodic retrieval activates the precuneus irrespective of the imagery content of word pair associates. Brain 1999, 122(2):255-263. 10.1093/brain/122.2.255.
40. Kriegeskorte N., Goebel R., Bandettini P. Information-based functional brain mapping. Proc. Natl. Acad. Sci. U. S. A. 2006, 103:3863-3868.
41. Kriegeskorte N., Mur M., Bandettini P. Representational similarity analysis - Connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2008, 2(4):1-28.
42. Kriegeskorte N., Mur M., Ruff D.A., Kiani R., Bodurka J., Esteky H., Tanaka K., Bandettini P.A. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 2008, 60:1126-1141.
43. Landauer T., Dumais S. A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol. Rev. 1997, 104(2):211-240.
44. Lazebnik S., Schmid C., Ponce J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. Proceedings of CVPR 2006, 2169-2178. IEEE, Washington.
45. Lee S.H., Kravitz D.J., Baker C.I. Disentangling visual imagery and perception of real-world objects. NeuroImage 2012, 59:4064-4073.
46. Leeds D.D., Seibert D.A., Pyles J.A., Tarr M.J. Comparing visual representations across human fMRI and computational vision. J. Vis. 2013, 13(13):1-27. (25).
47. Lenci A. Distributional semantics in linguistic and cognitive research. Ital. J. Linguist. 2008, 20(1):1-31.
48. Louwerse M. Embodied representations are encoded in language. Psychon. Bull. Rev. 2008, 15:838-844.
49. Louwerse M.M., Hutchinson S. Neurological evidence linguistic processes precede perceptual simulation in conceptual processing. Front. Psychol. 2012, 3:385. 10.3389/fpsyg.2012.00385.
50. Lowe D. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60(2):91-110.
51. Lund K., Burgess C. Producing high-dimensional semantic spaces from lexical co-occurrence. Behav. Res. Methods Instrum. Comput. 1996, 28:203-208.
52. Mahon B.Z., Caramazza A. Concepts and categories: a cognitive neuropsychological perspective. Annu. Rev. Psychol. 2009, 60:27-51.
53. Manning C., Schütze H. Foundations of statistical natural language processing 1999, MIT press.
54. Manning C., Raghavan P., Schütze H. Introduction to information retrieval 2008, Cambridge University Press, Cambridge, UK.
55. Martin A. The representation of object concepts in the brain. Annu. Rev. Psychol. 2007, 58:25-45.
56. Mitchell T.M., Shinkareva S.V., Carlson A., Chang K.-M., Malave V.L., Mason R.A., Just M.A. Predicting human brain activity associated with the meaning of nouns. Science 2008, 320:1191-1195.
57. Murphy B., Talukdar P., Mitchell T. Selecting corpus-semantic models for neurolinguistic decoding. Proceedings of First Joint Conference on Lexical and Computational Semantics (*SEM) 2012, 114.
58. O'Craven K., Downing P., Kanwisher N. fMRI evidence for objects as the units of attentional selection. Nature 1999, 401:584-587.
59. Op de Beeck H.P., Torfs K., Wagemans J. Perceived shape similarity among unfamiliar objects and the organization of the human object vision pathway. J. Neurosci. 2008, 28:10111-10123.
60. Paivio A. Imagery and verbal processes 1971, Holt, Rinehart, and Winston, New York.
61. Peelen M.V., Fei-fei L., Kastner S. Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature 2009, 460:94-97.
62. Pulvermüller F. How neurons make meaning: brain mechanisms for embodied and abstract-symbolic semantics. Trends Cogn. Sci. 2013, 17(9):458-470.
63. Pulvermüller F., Cooper-Pye E., Dine C., Hauk O., Nestor P.J., Patterson K. The word processing deficit in semantic dementia: all categories are equal, but some categories are more equal than others. J. Cogn. Neurosci. 2010, 22(9):2027-2041.
64. Raizada R.D.S., Connolly A.C. What makes different people's representations alike: neural similarity-space solves the problem of across-subject fMRI decoding. J. Cogn. Neurosci. 2012, 24(4):868-877.
65. Reddy L., Tsuchiya N., Serre T. Reading the mind's eye: decoding category information during mental imagery. NeuroImage 2010, 50:818-825.
66. Serre T., Oliva A., Poggio T. A feedforward architecture accounts for rapid categorization. Proc. Natl. Acad. Sci. U. S. A. 2007, 104:6424-6429.
67. Sivic J., Zisserman A. Video Google: A text retrieval approach to object matching in videos. Proceedings of ICCV 2003, 1470-1477. IEEE, Nice, France.
68. Steiger J.H. Tests for comparing elements of a correlation matrix. Psychol. Bull. 1980, 87(2):245-251.
69. Stokes M., Thompson R., Cusack R., Duncan J. Top-down activation of shape-specific population codes in visual cortex during mental imagery. J. Neurosci. 2009, 29:1565-1572.
70. Stokes M., Thompson R., Nobre A.C., Duncan J. Shape-specific preparatory activity mediates attention to targets in human visual cortex. Proc. Natl. Acad. Sci. U. S. A. 2009, 106:19569-19574.
71. Stokes M., Saraiva A., Rohenkohl G., Nobre A.C. Imagery for shapes activates position-invariant representations in human visual cortex. NeuroImage 2011, 56:1540-1545.
72. Trumpp N.M., Kliese D., Hoenig K., Haarmeier T., Kiefer M. Losing the sound of concepts: Damage to auditory association cortex impairs the processing of sound-related concepts. Cortex 2013, 49(2):474-486.
73. Turney P., Pantel P. From frequency to meaning: Vector space models of semantics. J. Artif. Intell. Res. 2010, 37:141-188.
74. Tzourio-Mazoyer N., Landeau B., Papathanassiou D., Crivello F., Etard O., Delcroix N., Mazoyer B., Joliot M. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. NeuroImage 2002, 15:273-289.
75. Yanulevskaya V., Uijlings J., Bruni E., Sartori A., Zamboni E., Bacci F., Melcher D., Sebe N. In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. Proceedings of the 20th ACM international conference on Multimedia 2012, 349-358. ACM.
76. Zwaan R.A., Stanfield R.A., Yaxley R.H. Language comprehenders mentally represent the shapes of objects. Psychol. Sci. 2002, 13(2):168-171.