Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation

PLoS Computational Biology

Volumn 10, Issue 11, 2014, Pages

  Khaligh Razavi, Seyed Mahdi ^a   Kriegeskorte, Nikolaus ^a  

^a MEDICAL RESEARCH COUNCIL   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATION THEORY; COMPUTATIONAL GEOMETRY; COMPUTATIONAL METHODS; CONVOLUTION; MAMMALS; NEUROPHYSIOLOGY; OBJECT RECOGNITION;

COMPUTATIONAL MODELLING; COMPUTATIONAL OBJECTS; DISSIMILARITY MATRIX; INFERIOR TEMPORAL CORTICES; MODEL REPRESENTATION; NON-HUMAN PRIMATE; PERFORMANCE; TEMPORAL DATA; VISION MODEL; VISUAL OBJECT RECOGNITION;

DEEP NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; BRAIN FUNCTION; BRAIN REGION; COMPUTER MODEL; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GEOMETRY; HISTOGRAM; HUMAN; HUMAN VERSUS ANIMAL COMPARISON; IMAGE ANALYSIS; INFERIOR TEMPORAL CORTEX; INTERMETHOD COMPARISON; NONHUMAN; PATTERN RECOGNITION; SUPPORT VECTOR MACHINE; VALIDATION PROCESS; ANIMAL; BIOLOGICAL MODEL; BIOLOGY; HAPLORHINI; PHYSIOLOGY; PROCEDURES; TEMPORAL LOBE;

ANIMALS; COMPUTATIONAL BIOLOGY; HAPLORHINI; HUMANS; MODELS, NEUROLOGICAL; SUPPORT VECTOR MACHINES; TEMPORAL LOBE;

EID: 84912102994     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1003915     Document Type: Article

References (91)

1. Desimone R, Albright TD, Gross CG, Bruce C, (1984) Stimulus-selective properties of inferior temporal neurons in the macaque. J Neurosci 4: 2051–2062.
2. Gross CG, (1994) How Inferior Temporal Cortex Became a Visual Area. Cereb Cortex 4: 455–469 doi:10.1093/cercor/4.5.455
3. Tanaka K, (1996) Inferotemporal Cortex and Object Vision. Annu Rev Neurosci 19: 109–139 doi:10.1146/annurev.ne.19.030196.000545
4. Hung CP, Kreiman G, Poggio T, DiCarlo JJ, (2005) Fast Readout of Object Identity from Macaque Inferior Temporal Cortex. Science 310: 863–866 doi:10.1126/science.1117593
5. Zoccolan D, Kouh M, Poggio T, DiCarlo JJ, (2007) Trade-Off between Object Selectivity and Tolerance in Monkey Inferotemporal Cortex. J Neurosci 27: 12292–12307 doi:10.1523/JNEUROSCI.1897-07.2007
6. Kiani R, Esteky H, Mirpour K, Tanaka K, (2007) Object Category Structure in Response Patterns of Neuronal Population in Monkey Inferior Temporal Cortex. J Neurophysiol 97: 4296–4309 doi:10.1152/jn.00024.2007
7. Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, et al. (2008) Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey. Neuron 60: 1126–1141 doi:10.1016/j.neuron.2008.10.043
8. Sato T, Uchida G, Tanifuji M, (2009) Cortical Columnar Organization Is Reconsidered in Inferior Temporal Cortex. Cereb Cortex 19: 1870–1888 doi:10.1093/cercor/bhn218
9. Bell AH, Hadj-Bouziane F, Frihauf JB, Tootell RBH, Ungerleider LG, (2009) Object Representations in the Temporal Cortex of Monkeys and Humans as Revealed by Functional Magnetic Resonance Imaging. J Neurophysiol 101: 688–700 doi:10.1152/jn.90657.2008
10. Mur M, Bodurka J, Goebel R, Bandettini PA, Kriegeskorte N, (2013) Human object-similarity judgments reflect and transcend the primate-IT object representation. Front Psychol 4: 128 doi:10.3389/fpsyg.2013.00128
11. Carlson T, Tovar DA, Alink A, Kriegeskorte N, (2013) Representational dynamics of object vision: The first 1000 ms. J Vis 13: 1 doi:10.1167/13.10.1
12. Cichy RM, Pantazis D, Oliva A, (2014) Resolving human object recognition in space and time. Nat Neurosci 17: 455–462 doi:10.1038/nn.3635
13. Majaj N, Hong H, Solomon E, DiCarlo J (2012) A unified neuronal population code fully explains human object recognition Cosyne 2012.
14. Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO, et al. (2012) The Representation of Biological Classes in the Human Brain. J Neurosci 32: 2608–2618 doi:10.1523/JNEUROSCI.5547-11.2012
15. Naselaris T, Stansbury DE, Gallant JL, (n.d.) Cortical representation of animate and inanimate objects in complex natural scenes. Journal of Physiology-Paris Available: http://www.sciencedirect.com/science/article/pii/S092842571200006X. Accessed 20 September 2012.
16. Mur M, Ruff DA, Bodurka J, De Weerd P, Bandettini PA, et al. (2012) Categorical, Yet Graded – Single-Image Activation Profiles of Human Category-Selective Cortical Regions. J Neurosci 32: 8649–8662 doi:10.1523/JNEUROSCI.2334-11.2012
17. Kriegeskorte N, (2009) Relating population-code representations between man, monkey, and computational models. Front Neurosci 3: 363–73 Available: http://www.frontiersin.org/neuroscience/10.3389/neuro.01.035.2009/pdf/full. Accessed 18 January 2012.
18. Kriegeskorte N, Mur M, Bandettini P, (2008) Representational similarity analysis – connecting the branches of systems neuroscience. Front Syst Neurosci 2: 4 doi:10.3389/neuro.06.004.2008
19. Leeds DD, Seibert DA, Pyles JA, Tarr MJ, (2013) Comparing visual representations across human fMRI and computational vision. J Vis 13: 25 doi:10.1167/13.13.25
20. Serre T, Oliva A, Poggio T, (2007) A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Sciences 104: 6424–6429 doi:10.1073/pnas.0700622104
21. Riesenhuber M, Poggio T, (1999) Hierarchical models of object recognition in cortex. nature neuroscience 2: 1019–1025.
22. Huth AG, Nishimoto S, Vu AT, Gallant JL, (2012) A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain. Neuron 76: 1210–1224 doi:10.1016/j.neuron.2012.10.014
23. Baldassi C, Alemi-Neissi A, Pagan M, DiCarlo JJ, Zecchina R, et al. (2013) Shape Similarity, Better than Semantic Membership, Accounts for the Structure of Visual Object Representations in a Population of Monkey Inferotemporal Neurons. PLoS Comput Biol 9: e1003167 doi:10.1371/journal.pcbi.1003167
24. Khaligh-Razavi S-M (2014) What you need to know about the state-of-the-art computational models of object-vision: A tour through the models. arXiv:14072776 [cs, q-bio]. Available: http://arxiv.org/abs/1407.2776. Accessed 11 July 2014.
25. Pillow JW, Shlens J, Paninski L, Sher A, Litke AM, et al. (2008) Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature 454: 995–999 doi:10.1038/nature07140
26. Mitchell TM, Shinkareva SV, Carlson A, Chang K-M, Malave VL, et al. (2008) Predicting Human Brain Activity Associated with the Meanings of Nouns. Science 320: 1191–1195 doi:10.1126/science.1152876
27. Kay KN, Naselaris T, Prenger RJ, Gallant JL, (2008) Identifying natural images from human brain activity. Nature 452: 352–355 doi:10.1038/nature06713
28. Dumoulin SO, Wandell BA, (2008) Population receptive field estimates in human visual cortex. NeuroImage 39: 647–660 doi:10.1016/j.neuroimage.2007.09.034
29. Kriegeskorte N, Kievit RA, (2013) Representational geometry: integrating cognition, computation, and the brain. Trends in Cognitive Sciences 17: 401–412 doi:10.1016/j.tics.2013.06.007
30. Nili H, Wingfield C, Walther A, Su L, Marslen-Wilson W, et al. (2014) A Toolbox for Representational Similarity Analysis. PLoS Comput Biol 10: e1003553 doi:10.1371/journal.pcbi.1003553
31. Ganguli S, Sompolinsky H, (2012) Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. Annu Rev Neurosci 35: 485–508 doi:10.1146/annurev-neuro-062111-150410
32. Johnson WB, Lindenstrauss J, (1984) Extensions of Lipschitz mappings into a Hilbert space. Contemporary mathematics 26: 1.
33. Mutch J, Lowe DG, (2008) Object Class Recognition and Localization Using Sparse Features with Limited Receptive Fields. International Journal of Computer Vision 80: 45–57 doi:10.1007/s11263-007-0118-0
34. Wallis G, Rolls ET, (1997) A model of invariant object recognition in the visual system. Prog Neurobiol 51: 167–194.
35. Ghodrati M, Khaligh-Razavi S-M, Ebrahimpour R, Rajaei K, Pooyan M, (2012) How Can Selection of Biologically Inspired Features Improve the Performance of a Robust Object Recognition Model? PLoS ONE 7: e32357 doi:10.1371/journal.pone.0032357
36. Rajaei K, Khaligh-Razavi S-M, Ghodrati M, Ebrahimpour R, Shiri Ahmad Abadi ME, (2012) A Stable Biologically Motivated Learning Mechanism for Visual Feature Extraction to Handle Facial Categorization. PLoS ONE 7: e38478 doi:10.1371/journal.pone.0038478
37. Ghodrati M, Farzmahdi A, Rajaei K, Ebrahimpour R, Khaligh-Razavi S-M, (2014) Feedforward Object-Vision Models Only Tolerate Small Image Variations Compared to Human. Frontiers in Computational Neuroscience 8 doi:10.3389/fncom.2014.00074
38. Sountsov P, Santucci DM, Lisman JE, (2011) A biologically plausible transform for visual recognition that is invariant to translation, scale, and rotation. Frontiers in computational neuroscience 5: 53.
39. Jarrett K, Kavukcuoglu K, Ranzato MA, LeCun Y (2009) What is the best multi-stage architecture for object recognition? Computer Vision, 2009 IEEE 12th International Conference on. pp. 2146–2153.
40. LeCun Y, Bengio Y, (1995) Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361.
41. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet Classification with Deep Convolutional Neural Networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ, editors. Advances in Neural Information Processing Systems 25. Curran Associates, Inc. pp. 1097–1105.
42. Oliva A, Torralba A, (2001) Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope. International Journal of Computer Vision 42: 145–175.
43. Lowe DG (1999) Object recognition from local scale-invariant features. iccv. p. 1150.
44. Lazebnik S, Schmid C, Ponce J (2006) Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. Vol. 2. pp. 2169–2178. doi:10.1109/CVPR.2006.68.
45. Bosch A, Zisserman A, Munoz X (2007) Representing shape with a spatial pyramid kernel. Proceedings of the 6th ACM international conference on Image and video retrieval. CIVR '07. New York, NY, USA: ACM. pp. 401–408. Available: http://doi.acm.org/10.1145/1282280.1282340. Accessed 6 April 2012.
46. Shechtman E, Irani M (2007) Matching Local Self-Similarities across Images and Videos. IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR '07. pp. 1–8. doi:10.1109/CVPR.2007.383198.
47. Deselaers T, Ferrari V (2010) Global and efficient self-similarity for object classification and detection. Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. pp. 1633–1640. doi:10.1109/CVPR.2010.5539775.
48. Ojala T, Pietikäinen M, Mäenpää T, (2001) A generalized local binary pattern operator for multiresolution gray scale and rotation invariant texture classification. Advances in Pattern Recognition—ICAPR 2001: 399–408.
49. Deng J, Dong W, Socher R, Li L-J, Li K, et al. (2009) ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009. pp. 248–255. doi:10.1109/CVPR.2009.5206848.
50. Efron B, Tibshirani R, (1986) Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. Statist Sci 1: 54–75 doi:10.1214/ss/1177013815
51. Lawson CL, Hanson RJ (1974) Solving least squares problems. Englewood Cliffs, NJ: Prentice-hall. Vol. 161.
52. Krizhevsky A., Sutskever I. and Hinton, G E. (2012) ImageNet Classification with Deep Convolutional Neural Networks. NIPS. Lake Tahoe, Nevada.
53. Konkle T, Oliva A, (2012) A Real-World Size Organization of Object Responses in Occipitotemporal Cortex. Neuron 74: 1114–1124 doi:10.1016/j.neuron.2012.04.036
54. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, et al. (2013) DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. arXiv:13101531 [cs]. Available: http://arxiv.org/abs/1310.1531. Accessed 7 May 2014.
55. Oliva A, Torralba A (2006) Building the gist of a scene: the role of global image features in recognition. Progress in Brain Research. p. 2006.
56. Belongie S, Malik J, Puzicha J, (2002) Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 509–522.
57. Berg AC, Berg TL, Malik J (2005) Shape matching and object recognition using low distortion correspondences. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. pp. 26–33.
58. Zhang H, Berg AC, Maire M, Malik J (2006) SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition. Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. Vol. 2. pp. 2126–2136. doi:10.1109/CVPR.2006.301.
59. Vedaldi A, Gulshan V, Varma M, Zisserman A (2009) Multiple kernels for object detection. Computer Vision, 2009 IEEE 12th International Conference on. pp. 606–613. doi:10.1109/ICCV.2009.5459183.
60. Lowe DG, (2004) Distinctive Image Features from Scale-Invariant Keypoints. Int J Comput Vision 60: 91–110 doi:10.1023/B:VISI.0000029664.99615.94
61. Tromans JM, Harris M, Stringer SM, (2011) A Computational Model of the Development of Separate Representations of Facial Identity and Expression in the Primate Visual System. PLoS ONE 6: e25616 doi:10.1371/journal.pone.0025616
62. Stringer SM, Rolls ET, Tromans JM, (2007) Invariant object recognition with trace learning and multiple stimuli present during training. Network 18: 161–187 doi:10.1080/09548980701556055
63. Chatfield K, Philbin J, Zisserman A (2009) Efficient retrieval of deformable shape classes using local self-similarities. 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops). pp. 264–271. doi:10.1109/ICCVW.2009.5457691.
64. Ojala T, Pietikainen M, Maenpaa T, (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24: 971–987.
65. Pietikäinen M, (2010) Local Binary Patterns. Scholarpedia 5: 9775 doi:10.4249/scholarpedia.9775
66. HUBEL D, WIESEL T, (1962) Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology 160: 106–154.
67. Hubel DH, Wiesel TN, (1968) Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology 195: 215.
68. Zabbah S, Rajaei K, Mirzaei A, Ebrahimpour R, Khaligh-Razavi S-M, (2014) The impact of the lateral geniculate nucleus and corticogeniculate interactions on efficient coding and higher-order visual object processing. Vision Research 101: 82–93 doi:10.1016/j.visres.2014.05.006
69. Grossberg S, (1988) Adaptive pattern classification and universal recoding. I.: parallel development and coding of neural feature detectors 243–258.
70. Chang C-C, Lin C-J, (2011) LIBSVM: A library for support vector machines. ACM Trans Intell Syst Technol 2: 1–27 doi:10.1145/1961189.1961199
71. Hinton GE, Osindero S, Teh Y-W, (2006) A fast learning algorithm for deep belief nets. Neural computation 18: 1527–1554.
72. Bengio Y, (2009) Learning Deep Architectures for AI. Found Trends Mach Learn 2: 1–127 doi:10.1561/2200000006
73. Zeiler MD, Fergus R (2013) Visualizing and Understanding Convolutional Networks. arXiv:13112901 [cs]. Available: http://arxiv.org/abs/1311.2901. Accessed 26 March 2014.
74. Khaligh-Razavi S-M, Kriegeskorte N (2013) Object-vision models that better explain IT also categorize better, but all models fail at both. Cosyne Abstracts, Salt Lake City USA.
75. Yamins DLK, Hong H, Cadieu CF, Solomon EA, Seibert D, et al. (2014) Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc Natl Acad Sci USA 111: 8619–8624 doi:10.1073/pnas.1403112111
76. Cadieu CF, Hong H, Yamins DLK, Pinto N, Ardila D, et al. (2014) Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition. arXiv:14063284 [cs, q-bio]. Available: http://arxiv.org/abs/1406.3284. Accessed 17 July 2014.
77. Yamins DL, Hong H, Cadieu C, DiCarlo JJ (2013) Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, editors. Advances in Neural Information Processing Systems 26. Curran Associates, Inc. pp. 3093–3101.
78. DiCarlo JJ, Cox DD, (2007) Untangling invariant object recognition. Trends in Cognitive Sciences 11: 333–341.
79. Rust NC, DiCarlo JJ, (2010) Selectivity and Tolerance (“Invariance”) Both Increase as Visual Information Propagates from Cortical Area V4 to IT. J Neurosci 30: 12978–12995 doi:10.1523/JNEUROSCI.0179-10.2010
80. Kriegeskorte N, (2011) Pattern-information analysis: from stimulus decoding to computational-model testing. Neuroimage 56: 411–421 doi:10.1016/j.neuroimage.2011.01.061
81. Carlson TA, Simmons RA, Kriegeskorte N, Slevc LR, (2013) The Emergence of Semantic Meaning in the Ventral Temporal Pathway. Journal of Cognitive Neuroscience 1–12 doi:__10.1162/jocn_a_00458
82. Devereux BJ, Clarke A, Marouchos A, Tyler LK, (2013) Representational Similarity Analysis Reveals Commonalities and Differences in the Semantic Processing of Words and Objects. J Neurosci 33: 18906–18916 doi:10.1523/JNEUROSCI.3809-13.2013
83. Clarke A, Tyler LK, (2014) Object-specific semantic coding in human perirhinal cortex. J Neurosci 34 (14) 4766–75.
84. Biederman I, (1987) Recognition-by-components: A theory of human image understanding. Psychological Review 94: 115–147.
85. Stork DG (1989) Is backpropagation biologically plausible?, International Joint Conference on Neural Networks, 1989. IJCNN. pp. 241–246 vol. 2. doi:10.1109/IJCNN.1989.118705.
86. Földiák P, (1991) Learning Invariance from Transformation Sequences. Neural Computation 3: 194–200 doi:10.1162/neco.1991.3.2.194
87. Li N, DiCarlo JJ, (2010) Unsupervised Natural Visual Experience Rapidly Reshapes Size-Invariant Object Representation in Inferior Temporal Cortex. Neuron 67: 1062–1075 doi:10.1016/j.neuron.2010.08.029
88. Li N, DiCarlo JJ, (2012) Neuronal Learning of Invariant Object Representation in the Ventral Visual Stream Is Not Dependent on Reward. J Neurosci 32: 6611–6620 doi:10.1523/JNEUROSCI.3786-11.2012
89. Schultz J, Friston KJ, O'Doherty J, Wolpert DM, Frith CD, (2005) Activation in Posterior Superior Temporal Sulcus Parallels Parameter Inducing the Percept of Animacy. Neuron 45: 625–635 doi:10.1016/j.neuron.2004.12.052
90. Riesenhuber M, (2007) Appearance Isn't Everything: News on Object Representation in Cortex. Neuron 55: 341–344 doi:10.1016/j.neuron.2007.07.017
91. Mahon BZ, Milleville SC, Negri GAL, Rumiati RI, Caramazza A, et al. (2007) Action-Related Properties Shape Object Representations in the Ventral Stream. Neuron 55: 507–520 doi:10.1016/j.neuron.2007.07.011