Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis

Annual Review of Neuroscience

Volumn 35, Issue , 2012, Pages 485-508

  Ganguli, Surya ^a   Sompolinsky, Haim ^b,c  

^a STANFORD UNIVERSITY   (United States)

^b HEBREW UNIVERSITY OF JERUSALEM   (Israel)

^c HARVARD UNIVERSITY   (United States)

Author keywords

communication; connectomics; generalization; imaging; learning; memory; random projections

Indexed keywords

ALGORITHM; DATA ANALYSIS; GENE EXPRESSION; HUMAN; INFORMATION PROCESSING; LEARNING; MULTIFACTOR DIMENSIONALITY REDUCTION; NERVE CELL NETWORK; NERVE TRACT; NEUROSCIENCE; NONLINEAR SYSTEM; PRIORITY JOURNAL; REVIEW; SEMANTICS; SHORT TERM MEMORY; SYNAPSE;

BRAIN; DATA INTERPRETATION, STATISTICAL; HUMANS; LEARNING; MEMORY, SHORT-TERM; MENTAL PROCESSES; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); NEURONS; NONLINEAR DYNAMICS;

EID: 84862682946     PISSN: 0147006X     EISSN: 15454126     Source Type: Book Series    
DOI: 10.1146/annurev-neuro-062111-150410     Document Type: Review

References (106)

1. Aharon M, Elad M, Bruckstein A. 2006a. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Proc. 54(11):4311
2. Aharon M, Elad M, Bruckstein A. 2006b. On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them. Linear Algebr. Appl. 416(1):48-67 (Pubitemid 43737210)
3. Atick J. 1992. Could information theory provide an ecological theory of sensory processing? Netw. Comput. Neural Syst. 3(2):213-51
4. Atick J, Redlich A. 1992. What does the retina know about natural scenes? Neural Comput. 4(2):196-210
5. Baraniuk R. 2007. Compressive sensing. Signal Proc. Mag. IEEE 24(4):118-21
6. Baraniuk R. 2011. More is less: signal processing and the data deluge. Science 331(6018):717-19
7. Baraniuk R, Cevher V, Wakin M. 2010. Low-dimensional models for dimensionality reduction and signal recovery: a geometric perspective. Proc. IEEE 98(6):959-71
8. Baraniuk R, Davenport M, DeVore R, Wakin M. 2008. A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3):253-63
9. Baraniuk R, Wakin M. 2009. Random projections of smooth manifolds. Found. Comput. Math. 9(1):51-77
10. Barlow H. 1961. Possible principles underlying the transformation of sensory messages. In Sensory Communication, ed. WA Rosenblith, pp. 217-34. New York: Wiley
11. Barlow H. 2001. Redundancy reduction revisited. Netw. Comput. Neural Syst. 12(3):241-53 (Pubitemid 33626945)
12. Bayati M, Bento J, Montanari A. 2010. The LASSO risk: asymptotic results and real world examples. Neural Inf. Process. Syst. (NIPS)
13. Bell A, Sejnowski T. 1997. The independent components of natural scenes are edge filters. Vis. Res. 37(23):3327-38 (Pubitemid 27493806)
14. Blum A. 2006. Random projection, margins, kernels, and feature-selection. In Subspace, Latent Structure and Feature Selection, ed. C Saunders, MGrobelnik, S Gunn, J Shawe-Taylor, pp. 52-68. Heidelberg, Germ. : Springer (Pubitemid 44029879)
15. Borghuis B, Ratliff C, Smith R, Sterling P, Balasubramanian V. 2008. Design of a neuronal array. J. Neurosci. 28(12):3178-89 (Pubitemid 351442290)
16. Boyd S, Vandenberghe L. 2004. Convex Optimization. New York: Cambridge Univ Press
17. Bruckstein A, Donoho D, Elad M. 2009. From sparse solutions of systems of equations to sparse modeling of signals and images. Siam Rev. 51(1):34-81
18. Brunel N, Hakim V, Isope P, Nadal J, Barbour B. 2004. Optimal information storage and the distribution of synaptic weights: perceptron versus Purkinje cell. Neuron 43(5):745-57 (Pubitemid 39179732)
19. Burges C. 1998. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2(2):121-67 (Pubitemid 128695475)
20. Candes E, Plan Y. 2010. A probabilistic and RIPless theory of compressed sensing. IEEE Trans. Inf. Theory 57(11)7235-54
21. Candes E, Romberg J. 2007. Sparsity and incoherence in compressive sampling. Invers. Probl. 23(3):969-85 (Pubitemid 46838836)
22. Candes E, Romberg J, Tao T. 2006. Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8):1207-23 (Pubitemid 43988295)
23. Candes E, Tao T. 2005. Decoding by linear programming. IEEE Trans. Inf. Theory 51:4203-15 (Pubitemid 41800353)
24. Candes E, TaoT. 2006. Near-optimal signal recovery from random projections: universal encoding strategies? IEEE Trans. Inf. Theory 52(12):5406-25 (Pubitemid 46445381)
25. Candes E, Wakin M. 2008. An introduction to compressive sampling. IEEE Sig. Proc. Mag. 25(2):21-30
26. Coskun A, Sencan I, Su T, Ozcan A. 2010. Lensless wide-field fluorescent imaging on a chip using compressive decoding of sparse objects. Opt. Express 18(10):10510-23
27. Coulter W, Hillar C, Isley G, Sommer F. 2010. Adaptive compressed sensing-a new class of self-organizing coding models for neuroscience. Presented at IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 5494-97
28. Dai W, Sheikh M, Milenkovic O, Baraniuk R. 2009. Compressive sensing DNA microarrays. EURASIP J. Bioinf. Syst. Biol. 2009:162824
29. Dasgupta S, Gupta A. 2003. An elementary proof of a theorem of Johnson and Lindenstrauss. Random Struct. Algorithms 22(1):60-65
30. Daubechies I, Roussos E, Takerkart S, Benharrosh M, Golden C, et al. 2009. Independent component analysis for brain fMRI does not select for independence. Proc. Natl. Acad. Sci. 106(26):10415-20
31. Davenport M, Duarte M, Wakin M, Laska J, Takhar D, et al. 2007. The smashed filter for compressive classification and target recognition. Proc. Comput. Imaging V SPIE Electron Imaging, San Jose, CA
32. Dayan P, Abbott L. 2001. Theoretical Neuroscience. Computational andMathematicalModelling of Neural Systems. Cambridge, MA: MIT Press
33. Donoho D. 2000. High-dimensional data analysis: the curses and blessings of dimensionality. AMS Math Challenges Lecture, pp. 1-32
34. Donoho D. 2006. Compressed sensing. IEEE Trans. Inf. Theory 52(4):1289-306
35. Donoho D, Maleki A, Montanari A. 2009. Message-passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. USA 106(45):18914-19
36. Donoho D, Tanner J. 2005a. Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. USA 102:9452-57 (Pubitemid 40981702)
37. Donoho D, Tanner J. 2005b. Sparse nonnegative solution of underdetermined linear equations by linear programming. Proc. Natl. Acad. Sci. USA 102:9446-51 (Pubitemid 40981701)
38. Duarte M, DavenportM, TakharD, Laska J, Sun T, et al. 2008. Single-pixel imaging via compressive sampling. Signal Proc. Mag. IEEE 25(2):83-91
39. Duarte M, Davenport M, Wakin M, Baraniuk R. 2006. Sparse signal detection from incoherent projections. Proc. Acoust. Speech Signal Process. (ICASSP) 3:III-III
40. Duarte M, Davenport M, Wakin M, Laska J, Takhar D, et al. 2007. Multiscale random projections for compressive classification. Presented at IEEE Int. Conf. Image Process (ICIP) Int. Conf. 6:VI161-64, San Antonio, TX
41. Efron B, Hastie T, Johnstone I, Tibshirani R. 2004. Least angle regression. Ann. Stat. 32(2):407-99 (Pubitemid 41250302)
42. Engel A, den Broeck CV. 2001. Statistical Mechanics of Learning. London: Cambridge Univ. Press
43. Ganguli S, Bisley J, Roitman J, Shadlen M, Goldberg M, Miller K. 2008a. One-dimensional dynamics of attention and decision making in lip. Neuron 58(1):15-25 (Pubitemid 351467033)
44. Ganguli S, Huh D, Sompolinsky H. 2008b. Memory traces in dynamical systems. Proc. Natl. Acad. Sci. USA 105(48):18970-74
45. Ganguli S, Sompolinsky H. 2010a. Short-term memory in neuronal networks through dynamical compressed sensing. Neural Inf. Process. Syst. (NIPS) 23:667-75
46. Ganguli S, SompolinskyH. 2010b. Statistical mechanics of compressed sensing. Phys. Rev. Lett. 104(18):188701
47. Gardner E. 1988. The space of interactions in neural network models. J. Phys. A 21:257-70
48. Haupt J, Castro R, Nowak R, Fudge G, Yeh A. 2006. Compressive sampling for signal classification. Presented at Conf. Signals, Syst. Comput. (ACSSC), 40th, Asilomar, pp. 1430-34
49. Hegde C, Wakin M, Baraniuk R. 2007. Random projections for manifold learning. Neural Inf. Process. Syst. http://books. nips.cc/papers/files/nips20/ NIPS2007-1100. pdf
50. Hillar CJ, Sommer FT. 2011. Ramsey theory reveals the conditions when sparse coding on subsampled data is unique. ArXiv abs/1106. 3616
51. Hodgkin A, Huxley A. 1952. A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. 117:500-44
52. Hopfield J. 1982. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79(8):2554-59
53. Hu T, Chklovskii D. 2009. Reconstruction of sparse circuits using multi-neuronal excitation (rescume). Adv. Neural Inf. Proc. Syst. 22:790-98
54. Hu T, Druckmann S, Chklovskii D. 2011. Early sensory processing as predictive coding: subtracting sparse approximations by circuit dynamics. Front. Neurosci. Conf. Abs: COSYNE
55. Hyvarinen A. 2010. Statistical models of natural images and cortical visual representation. Top. Cogn. Sci. 2:251-64
56. Indyk P, Motwani R. 1998. Approximate nearest neighbors: towards removing the curse of dimensionality. Proc. Annu. ACM Symp. Theory Comput. , 30th, pp. 604-13
57. Isely G, Hillar CJ, Sommer FT. 2010. Deciphering subsampled data: adaptive compressive sampling as a principle of brain communication. Adv. Neural Inf. Proc. Syst. (NIPS) 23:910-18
58. Jaeger H. 2001. Short term memory in echo state networks. GMD Rep. 152. Germ. Natl. Res. Cent. Inf. Technol. , Bremen
59. Jaeger H, Haas H. 2004. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667):78-81
60. Johnson W, Lindenstrauss J. 1984. Extensions of Lipschitz mappings into a Hilbert space. Contemp. Math. 26:189-206
61. Kabashima Y, Wadayama T, Tanaka T. 2009. A typical reconstruction limit for compressed sensing based on l p-norm minimization. J. Stat. Mech. L09003
62. Kelly R, Smith M, Kass R, Lee T. 2010. Accounting for network effects in neuronal responses using l1 regularized point process models. Neural Inf. Proc. Syst. (NIPS) 23:1099-107
63. Kiani R, Esteky H, Mirpour K, Tanaka K. 2007. Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J. Neurophysiol. 97(6):4296-309 (Pubitemid 47080657)
64. Koulakov A, Rinberg D. 2011. Sparse incomplete representations: a novel role for olfactory granule cells. Neuron 72(1):124-36
65. Kreutz-Delgado K, Murray J, Rao B, Engan K, Lee T, Sejnowski T. 2003. Dictionary learning algorithms for sparse representation. Neural Comput. 15(2):349-96 (Pubitemid 37049825)
66. Kriegeskorte N, Mur M, Ruff D, Kiani R, Bodurka J, et al. 2008. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60(6):1126-41
67. Lage-Castellanos A, Pagnani A, Weigt M. 2009. Statistical mechanics of sparse generalization and graphical model selection. J. Stat. Mech. : Theory Exp. 2009:P10009
68. Lee S, Ganapathi V, KollerD. 2006a. Efficient structure learning ofMarkov networks using L1 regularization. Neural Inf. Process. Syst. (NIPS) 19:817-24
69. Lee S, Lee H, Abbeel P, Ng A. 2006b. Efficient l1 regularized logistic regression. Proc. Natl. Conf. on Artif. Intell. 21:401
70. Lewicki M. 2002. Efficient coding of natural sounds. Nat. Neurosci. 5(4):356-63 (Pubitemid 34279816)
71. Linsker R. 1990. Perceptual neural organization: some approaches based on network models and information theory. Annu. Rev. Neurosci. 13(1):257-81 (Pubitemid 20100651)
72. LustigM, Donoho D, Pauly J. 2007. Sparse MRI: the application of compressed sensing for rapidMRimaging. Magn. Reson. Med. 58(6):1182-95 (Pubitemid 350234205)
73. Lustig M, Donoho D, Santos J, Pauly J. 2008. Compressed sensing MRI. Signal Proc. Mag. IEEE 25(2):72-82
74. Maass W, Natschlager T, Markram H. 2002. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11):2531-60
75. MachensC, Romo R, Brody C. 2010. Functional, but not anatomical, separation of what andwhen in prefrontal cortex. J. Neurosci. 30(1):350-60
76. Marr D. 1969. A theory of cerebellar cortex. J. Physiol. 202(2):437-70
77. Mishchenko Y. 2011. Reconstruction of complete connectivity matrix for connectomics by sampling neural connectivity with fluorescent synaptic markers. J. Neurosci. Methods 196(2):289-302
78. Olshausen B, Field DJ. 1996a. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607-9 (Pubitemid 26177476)
79. Olshausen B, Field D. 1996b. Natural image statistics and efficient coding. Netw. Comput. Neural Syst. 7(2):333-39 (Pubitemid 126714799)
80. Olshausen B, Field D. 1997. Sparse coding with an overcomplete basis set: a strategy employed by v1? Vis. Res. 37(23):3311-25 (Pubitemid 27493805)
81. Ostojic S, Brunel N. 2011. From spiking neuron models to linear-nonlinear models. PLoS Comput. Biol. 7(1):e1001056
82. Parrish T, Hu X. 1995. Continuous update with random encoding (cure): a new strategy for dynamic imaging. Magn. Reson. Med. 33(3):326-36
83. Perrinet L. 2010. Role of homeostasis in learning sparse representations. Neural Comput. 22(7):1812-36
84. Rajan K, Abbott L, Sompolinsky H. 2010. Stimulus-dependent suppression of chaos in recurrent neural networks. Phys. Rev. E 82(1):011903
85. Rogers T, McClelland J. 2004. Semantic Cognition: A Parallel Distributed Processing Approach. Cambridge, MA: MIT Press
86. Rosenblatt F. 1958. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65(6):386-408
87. Rozell C, Johnson D, Baraniuk R, Olshausen B. 2008. Sparse coding via thresholding and local competition in neural circuits. Neural Comput. 20(10):2526-63
88. Seung H, Sompolinsky H, Tishby N. 1992. Statistical mechanics of learning from examples. Phys. Rev. A 45(8):6056-91
89. Simoncelli E, Olshausen B. 2001. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24(1):1193-216 (Pubitemid 32695258)
90. Smola A. 2000. Advances in Large Margin Classifiers. Cambridge, MA: MIT Press
91. Sussillo D, Abbott L. 2009. Generating coherent patterns of activity from chaotic neural networks. Neuron 63(4):544-57
92. Takhar D, Laska J, Wakin M, Duarte M, Baron D, et al. 2006. A new compressive imaging camera architecture using optical-domain compression. In Proc. Comput. Imaging IV Imaging, San Jose, CA
93. Taraska J, Zagotta W. 2010. Fluorescence applications in molecular neurobiology. Neuron 66(2):170-89
94. Tarifi M, Sitharam M, Ho J. 2011. Learning hierarchical sparse representations using iterative dictionary learning and dimension reduction. ArXiv 1106:0357
95. Taubman D, Marcellin M, Rabbani M. 2002. Jpeg2000: image compression fundamentals, standards and practice. J. Electron. Imaging 11:286
96. Tibshirani R. 1996. Regression shrinkage and selection via the LASSO. J. R. Stat. Soc. Ser. B (Methodol. ) 58:267-88
97. vanHateren J, Ruderman D 1998 Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex. Proc. R. Soc. Lond. Ser. B: Biol. Sci. 265 1412 2315-20 (Pubitemid 28561175)
98. vanHateren J, van der Schaaf A. 1998. Independent component filters of natural images compared with simple cells in primary visual cortex. Proc. R. Soc. Lond. Ser. B: Biol. Sci. 265(1394):359-66
99. Vapnik V. 1998. Statistical Learning Theory. New York:Wiley-Interscience
100. Wainwright M. 2009. Sharp thresholds for high-dimensional and noisy sparsity recovery using L1-constrained quadratic programming (LASSO). Inf. Theory IEEE Trans. 55(5):2183-202
101. Wainwright M, Ravikumar P, Lafferty J. 2007. High-dimensional graphical model selection using L1-regularized logistic regression. Adv. Neural Inf. Proc. Syst. 19:1465-72
102. Weiss Y, Chang H, Freeman W. 2007. Learning compressed sensing. Presented at Allerton Conf. , Urbana-Champaign, IL
103. White O, Lee D, Sompolinsky H. 2004. Short-term memory in orthogonal neural networks. Phys. Rev. Lett. 92(14):148102-5
104. Wilt B, Burns L, Ho E, Ghosh K, Mukamel E, Schnitzer M. 2009. Advances in light microscopy for neuroscience. Annu. Rev. Neurosci. 32:435-506
105. Yu B, Cunningham J, Santhanam G, Ryu S, Shenoy K, Sahani M. 2009. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. J. Neurophysiol. 102(1):614-35
106. Zhou S, Lafferty J, Wasserman L. 2009. Compressed and privacy-sensitive sparse regression. Inf. Theory IEEE Trans. 55(2):846-66