Generic decoding of seen and imagined objects using hierarchical visual features

Nature Communications

Volumn 8, Issue , 2017, Pages

  Horikawa, Tomoyasu ^a   Kamitani, Yukiyasu ^b  

^a ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

^b KYOTO UNIVERSITY   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL NEURAL NETWORK; BRAIN; COMPUTER VISION; GENETIC ANALYSIS; HOMOLOGY; IMAGE PROCESSING; MACHINERY;

BRAIN; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; INFORMATION RETRIEVAL; MACHINE; VISION; ADULT; ARTIFICIAL NEURAL NETWORK; FEMALE; GUIDED IMAGERY; IMAGE PROCESSING; IMAGINATION; MALE; PATTERN RECOGNITION; PHOTOSTIMULATION; PHYSIOLOGY; TIME FACTOR; YOUNG ADULT;

ADULT; BRAIN; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; IMAGERY (PSYCHOTHERAPY); IMAGINATION; MALE; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, VISUAL; PHOTIC STIMULATION; TIME FACTORS; VISUAL PERCEPTION; YOUNG ADULT;

EID: 85020240640     PISSN: None     EISSN: 20411723     Source Type: Journal    
DOI: 10.1038/ncomms15037     Document Type: Article

References (61)

1. Haxby, J. V. et al. Distributed and overlapping representations of faces and object in ventral temporal cortex. Science 293, 2425-2430 (2001).
2. Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679-685 (2005).
3. Harrison, S. A. & Tong, F. Decoding reveals the contents of visual working memory in early visual areas. Nature 458, 632-635 (2009).
4. Albers, A. M., Kok, P., Toni, I., Dijkerman, H. C. & De Lange, F. P. Shared representations for working memory and mental imagery in early visual cortex. Curr. Biol. 23, 1-5 (2013).
5. Xing, Y., Ledgeway, T., McGraw, P. V. & Schluppeck, D. Decoding working memory of stimulus contrast in early visual cortex. J. Neurosci. 33, 10301-10311 (2013).
6. Chadwick, M. J., Hassabis, D., Weiskopf, N. & Maguire, E. A. Decoding individual episodic memory traces in the human hippocampus. Curr. Biol. 20, 544-547 (2010).
7. Bonnici, H. M. et al. Detecting representations of recent and remote autobiographical memories in vmPFC and hippocampus. J. Neurosci. 32, 16982-16991 (2012).
8. Reddy, L., Tsuchiya, N. & Serre, T. Reading the mind's eye: decoding category information during mental imagery. Neuroimage 50, 818-825 (2010).
9. Cichy, R. M., Heinzle, J. & Haynes, J. D. Imagery and perception share cortical representations of content and location. Cereb. Cortex 22, 372-380 (2011).
10. Stokes, M., Thompson, R., Cusack, R. & Duncan, J. Top-down activation of shape-specific population codes in visual cortex during mental imagery. J. Neurosci. 29, 1565-1572 (2009).
11. Hassabis, D. et al. Imagine all the people: how the brain creates and uses personality models to predict behavior. Cereb. Cortex 24, 1979-1987 (2014).
12. Johnson, M. R. & Johnson, M. K. Decoding individual natural scene representations during perception and imagery. Front. Hum. Neurosci. 8, 1-14 (2014).
13. Horikawa, T., Tamaki, M., Miyawaki, Y. & Kamitani, Y. Neural decoding of visual imagery during sleep. Science 340, 639-642 (2013).
14. Kay, K. N., Naselaris, T., Prenger, R. J. & Gallant, J. L. Identifying natural images from human brain activity. Nature 452, 352-355 (2008).
15. Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M. & Gallant, J. L. Bayesian reconstruction of natural images from human brain activity. Neuron 63, 902-915 (2009).
16. Miyawaki, Y. et al. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60, 915-929 (2008).
17. Naselaris, T., Olman, C. A., Stansbury, D. E., Ugurbil, K. & Gallant, J. L. A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes. Neuroimage 105, 215-228 (2015).
18. Lowe, D. G. in Proceedings of the IEEE International Conference on Computer Vision, 1150-1157 (Washington, DC, USA (1999).
19. Csurka, G., Dance, C., Fan, L., Willarnowski, J. & Bray, C. Visual categorization with bags of keypoints. ECCV 1, 1-2 (2004).
20. Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1106-1114 (2012).
21. Riesenhuber, M. & Poggio, T. Hierarchical models of object recognition in cortex. Nat. Neurosci. 2, 1019-1025 (1999).
22. Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M. & Poggio, T. Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell. 29, 411-426 (2007).
23. Mutch, J. & Lowe, D. G. Object class recognition and localization using sparse features with limited receptive fields. Int. J. Comput. Vis. 80, 45-57 (2008).
24. Oliva, A. & Torralba, A. Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42, 145-175 (2001).
25. Cadieu, C. F. et al. Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS Comput. Biol. 10, e1003963 (2014).
26. Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619-8624 (2014).
27. Khaligh-Razavi, S. M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).
28. Güçlü, U. & van Gerven, M. A. J. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 100005-100014 (2015).
29. Rice, G. E., Watson, D. M., Hartley, T. & Andrews, T. J. Low-level image properties of visual objects predict patterns of neural response across category-selective regions of the ventral visual pathway. J. Neurosci. 34, 8837-8844 (2014).
30. Leeds, D. D., Seibert, D. A., Pyles, J. A. & Tarr, M. J. Comparing visual representations across human fMRI and computational vision. J. Vis. 13, 25 (2013).
31. Deng, J. et al. Imagenet: a large-scale hierarchical image database. IEEE CVPR (2009).
32. Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006).
33. Simonyan, K., Vedaldi, A. & Zisserman, A. in Visualising image classification models and saliency maps. ICLR Workshop (Banff, Canada, 2014).
34. Mahendran, A. & Vedaldi, A. Visualizing deep convolutional neural networks using natural pre-images. Int. J. Comput. Vis. 120, 233-255 (2016).
35. Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. in Deep Learning Workshop, ICML Conference (Lille, France, 2015).
36. Nguyen, A., Yosinski, J. & Clune, J. Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. arXiv:1602.03616 (2016).
37. Kobatake, E. & Tanaka, K. Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex. J. Neurophysiol. 71, 856-867 (1994).
38. Fellbaum, C. WordNet: An Electronic Lexical Database (MIT Press, 1998).
39. Larochelle, H., Erhan, D. & Bengio, Y. in AAAI Conference on Artificial Intelligence (Chicago, Illinois, USA, 2008).
40. Pearson, J. & Kosslyn, S. M. The heterogeneity of mental representation: ending the imagery debate. Proc. Natl Acad. Sci. USA 112, 10089-10092 (2015).
41. Pearson, J., Naselaris, T., Holmes, E. A. & Kosslyn, S. M. Mental imagery: functional mechanisms and clinical applications. Trends Cogn. Sci. 19, 590-602 (2015).
42. Ullman, S., Vidal-Naquet, M. & Sali, E. Visual features of intermediate complexity and their use in classification. Nat. Neurosci. 5, 682-687 (2002).
43. Fujita, I., Tanaka, K., Ito, M. & Cheng, K. Columns for visual features of objects in monkey inferotemporal cortex. Nature 360, 343-346 (1992).
44. Mitchell, T. M. et al. Predicting human brain activity associated with the meanings of nouns. Science 320, 1191-1195 (2008).
45. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).
46. Kriegeskorte, N. Relating population-code representations between man, monkey, and computational models. Front. Neurosci. 3, 363-373 (2009).
47. Kriegeskorte, N. & Kievit, R. A. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17, 401-412 (2013).
48. Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).
49. Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. arXiv: 1412.0035 (2014).
50. Xu, K. et al. Show, attend and tell: neural image caption generation with visual attention. arXiv: 1502.03044 (2015).
51. Engel, S. A. et al. fMRI of human visual cortex. Nature 369, 525 (1994).
52. Sereno, M. I. et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 268, 889-893 (1995).
53. Kourtzi, Z. & Kanwisher, N. Cortical regions involved in perceiving object shape. J. Neurosci. 20, 3310-3318 (2000).
54. Kanwisher, N., McDermott, J. & Chun, M. M. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17, 4302-4311 (1997).
55. Epstein, R. & Kanwisher, N. A cortical representation of the local visual environment. Nature 392, 598-601 (1998).
56. Hubel, D. & Wiesel, T. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 160, 106-154 (1962).
57. Hubel, D. & Wiesel, T. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. J. Neurophysiol. 28, 229-289 (1965).
58. Vedaldi, A. & Fulkerson, B. VLFeat: an open and portable library of computer vision algorithms. (version 0.9.9) (2008). http://www.vlfeat.org/.
59. Yamashita, O., Sato, M. A., Yoshioka, T., Tong, F. & Kamitani, Y. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. Neuroimage 42, 1414-1429 (2008).
60. Attias, H. in Proceedings of 15th Conference on Uncertainty in Artificial Intelligence, 21-30 (Stockholm, Sweden, 1999).
61. Sato, M. A. Online model selection based on the variational Bayes. Neural Comput. 13, 1649-1681 (2001).