Sparse coding of sensory inputs

Current Opinion in Neurobiology

Volumn 14, Issue 4, 2004, Pages 481-487

  Olshausen, Bruno A ^a   Field, David J ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b Department of Population Medicine and Diagnostic Sciences   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASSOCIATIVE MEMORY; BRAIN FUNCTION; CODING; HUMAN; MEMORY CONSOLIDATION; NERVE CELL; NERVE FUNCTION; NEUROPHYSIOLOGY; NONHUMAN; PRIORITY JOURNAL; REVIEW; SENSORY STIMULATION; SENSORY SYSTEM; SIGNAL TRANSDUCTION; SPARSE CODING; THEORY;

ACTION POTENTIALS; AFFERENT PATHWAYS; ANIMALS; BRAIN; HUMANS; MODELS, NEUROLOGICAL; NEURONS, AFFERENT; SENSATION; SIGNAL TRANSDUCTION; VISUAL CORTEX;

EID: 4043166218     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2004.07.007     Document Type: Review

References (54)

1. M.J. Berry, D.K. Warland, and M. Meister The structure and precision of retinal spike trains Proc Natl Acad Sci USA 94 1997 5411 5416
2. P. Reinagel How do visual neurons respond in the real world? Curr Opin Neurobiol 11 2001 437 442
3. D.J. Willshaw, O.P. Buneman, and H.C. Longuet-Higgins Non-holographic associative memory Nature 222 1969 960 962
4. G. Palm On associative memory Biol Cybern 36 1980 19 31
5. E.B. Baum, J. Moody, and F. Wilczek Internal representations for associative memory Biol Cybern 59 1988 217 228
6. P. Kanerva Sparse distributed memory and related models M.H. Hassoun Associative Neural Memories: Theory and Implementation 1993 Oxford University Press New York 50 76
7. C. Zetzsche Sparse coding: the link between low level vision and associative memory R. Eckmiller G. Hartmann G. Hauske Parallel Processing in Neural Systems and Computers 1990 Elsevier Science Amsterdam (North-Holland) 273 276
8. D.J. Field What is the goal of sensory coding? Neural Comput 6 1994 559 601
9. G. Palm, and F.T. Sommer Associative data storage and retrieval in neural networks E. Domany J.L. van Hemmen K. Schulten Models of Neural Networks III 1996 Springer New York 79 118
10. H.B. Barlow Single units and sensation: a neuron doctrine for perceptual psychology? Perception 1 1972 371 394
11. B.A. Olshausen Principles of image representation in visual cortex L.M. Chalupa J.S. Werner The Visual Neurosciences 2003 MIT Press Boston MA 1603 1615
12. D.J. Field Relations between the statistics of natural images and the response properties of cortical cells J Opt Soc Am A 4 1987 2379 2394
13. J.J. Atick Could information theory provide an ecological theory of sensory processing? Network 3 1992 213 251
14. J.H. van Hateren A theory of maximizing sensory information Biol Cybern 68 1992 23 29
15. D.L. Ruderman The statistics of natural images Network: Comput Neural Systems 5 1994 517 548
16. D.L. Ruderman Origins of scaling in natural images Vision Res 37 1997 3385 3398
17. E.P. Simoncelli, and B.A. Olshausen Natural image statistics and neural representation Annu Rev Neurosci 24 2001 1193 1215
18. E.P. Simoncelli Vision and the statistics of the visual environment Curr Opin Neurobiol 13 2003 144 149. The author presents a review discussing the theory of efficient coding, focusing on the visual system.
19. B.A. Olshausen, and D.J. Field Emergence of simple cell receptive field properties by learning a sparse code for natural images Nature 381 1996 607 609
20. A.J. Bell, and T.J. Sejnowski The 'independent components' of natural scenes are edge filters Vision Res 37 1997 3327 3338
21. J.H. van Hateren, and A. van der Schaaf Independent component filters of natural images compared with simple cells in primary visual cortex Proc R Soc Lond B Biol Sci 265 1998 359 366
22. J.H. van Hateren, and D.L. Ruderman Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex Proc R Soc Lond B Biol Sci 265 1998 2315 2320
23. A. Hyvarinen, and P.O. Hoyer Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces Neural Comput 12 2000 1705 1720
24. M.S. Lewicki Efficient coding of natural sounds Nat Neurosci 5 2002 356 363. The author uses the independent components analysis or sparse coding framework to argue that the properties of early auditory neurons are well suited to producing a sparse independent code for natural sounds. The results demonstrate interesting differences between optimal codes for animal communication and other environmental sounds.
25. B.A. Olshausen, and K.N. O'Connor A new window on sound Nat Neurosci 5 2002 292 294
26. A. Lee, K.S. Pedersen, and D. Mumford The nonlinear statistics of high-contrast patches in natural images Int J Comput Vis 54 2003 83 103. The authors provide a detailed analysis of the statistical regularities found in 3 x 3 pixel patches drawn from natural scenes. The resulting nine-dimensional state-space is not easily simplified, but the study demonstrates that much of the variance of the patches can be described in terms of lower dimensional manifolds.
27. S.T. Roweis, and L.K. Saul Nonlinear dimensionality reduction by locally linear embedding Science 290 2000 2323 2326
28. L. Wiskott, and L. Sejnowski Slow feature analysis: Unsupervised learning of invariances Neural Comput 14 2002 715 770. The authors describe a novel technique for learning the invariant structure of images based on using a non-linear transform to extract slowly-varying components in images.
29. C. Zetzsche, G. Krieger, and B. Wegmann The atoms of vision: Cartesian or polar? J Opt Soc Am A 16 1999 1554 1565
30. E.P. Simoncelli, W.T. Freeman, E.H. Adelson, and D.J. Heeger Shiftable multiscale transforms IEEE Transactions on Information Theory 38 1992 587 607
31. B.A. Olshausen, and D.J. Field Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Res 37 1997 3311 3325
32. M.S. Lewicki, and T.J. Sejnowski Learning overcomplete representations Neural Comput 12 2000 337 365
33. P. Foldiak Sparse coding in the primate cortex M.A. Arbib The Handbook of Brain Theory and Neural Networks 1995 MIT Press Boston MA 895 989
34. W.B. Levy, and R.A. Baxter Energy efficient neural codes Neural Comput 8 1996 531 543
35. D. Attwell, and S.B. Laughlin An energy budget for signaling in the grey matter of the brain J Cereb Blood Flow Metab 21 2001 1133 1145
36. P. Lennie The cost of cortical computation Curr Biol 13 2003 493 497. The author provides further evidence of an important metabolic reason for a sparse representation. The author calculates the cost of a neural spike and argues that these constraints require that the brain might have less than 1% neurons active at any given time.
37. R. Baddeley, L.F. Abbott, M.C. Booth, F. Sengpiel, T. Freeman, E.A. Wakeman, and E.T. Rolls Responses of neurons in primary and inferior temporal visual cortices to natural scenes Proc R Soc Lond B Biol Sci 264 1998 1775 1783
38. E.T. Rolls, and M.J. Tovee Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex J Neurophysiol 73 1995 713 726
39. W.E. Vinje, and J.L. Gallant Sparse coding and decorrelation in primary visual cortex during natural vision Science 287 2000 1273 1276
40. M. Weliky, J. Fiser, R.H. Hunt, and D.N. Wagner Coding of natural scenes in primary visual cortex Neuron 37 2003 703 718. The authors report that neural responses demonstrate high lifetime sparseness and population sparseness in response to natural images.
41. B. Willmore, and D.J. Tolhurst Characterizing the sparseness of neural codes Network 12 2001 255 270
42. W.E. Vinje, and J.L. Gallant Natural stimulation of the nonclassical receptive field increases information transmission efficiency in V1 J Neurosci 22 2002 2904 2915. It has long been known that stimulation outside of the classical receptive field of a neuron has a modulatory effect on its response to stimuli within the classical receptive field, but typically these effects have been explored through a reductionist approach using highly simplified stimuli. Here, the authors take a more ecologically motivated approach and show that when natural scenes are used as stimuli, the context sparsifies the response of the neuron - an effect not seen in previous studies.
43. M.P. Young, and S. Yamane Sparse population coding of faces in the inferotemporal cortex Science 256 1992 1327 1331
44. M. DeWeese, M. Wehr, and A. Zador Binary spiking in auditory cortex J Neurosci 23 2003 7940 7949. The authors show that in the auditory system some neurons have highly sparse responses. In response to tones, some neurons show a consistent response of just one spike.
45. M. Brecht, and B. Sakmann Dynamic representation of whisker deflection by synaptic potentials in spiny stellate and pyramidal cells in the barrels and septa of layer 4 rat somatosensory cortex J Physiol 543 2002 49 70. Through intracellular recording in rat somatosensory cortex, the authors show that outputs of layer 4 are computed from sparse and selective synaptic input.
46. J. Perez-Orive, O. Mazor, G.C. Turner, S. Cassenaer, R.I. Wilson, and G. Laurent Oscillations and sparsening of odor representations in the mushroom body Science 297 2002 359 365. The authors show that Kenyon cells in the mushroom bodies of insects provide a sparse code of odors.
47. G. Laurent Olfactory network dynamics and the coding of multidimensional signals Nat Rev Neurosci 3 2002 884 895. The author presents an in-depth discussion of the cellular mechanisms thought to underlie sparse coding in the insect olfactory system, how it could be derived from oscillatory patterns of activity known to occur at earlier stages of processing, and its relation to associative memory models.
48. •].
49. L.T. Thompson, and P.J. Best Place cells and silent cells in the hippocampus of freely-behaving rats J Neurosci 9 1989 2382 2390
50. C.A. Barnes, W.E. Skaggs, B.L. McNaughton, M.L. Haworth, M. Permenter, M. Archibeque, and Erickson CA Chronic recording of neuronal populations in the temporal lobe of awake young adult and geriatric primates [abstract] Soc Neurosci 2003 Program no. 518.8
51. I.N. Beloozerova, M.G. Sirota, and H.A. Swadlow Activity of different classes of neurons of the motor cortex during locomotion J Neurosci 23 2003 1087 1097
52. M. Brecht, M. Schneider, B. Sakmann, and T.W. Margrie Whisker movements evoked by stimulation of single pyramidal cells in rat motor cortex Nature 427 2004 704 710. Contrary to the 'mass-action' hypothesis, the authors show that stimulation of a single neuron in motor cortex is sufficient to evoke whisker movements in a rat.
53. Hahnloser RHR, A. Kozhevnikov, and M.S. Fee An ultrasparse code underlies the generation of neural sequences in songbirds Nature 419 2002 65 70. The authors show that neurons in nucleus HVC (high vocal center) of the zebra finch emit one spike at a precise moment in time during the production of a song.
54. I.R. Fiete, R.H. Hahnloser, M.S. Fee, and H.S. Seung Temporal sparseness of the premotor drive is important for rapid learning in a neural network model of birdsong J Neurophysiol 2004 (Epub ahead of print)