Sparse and silent coding in neural circuits

Neurocomputing

Volumn 79, Issue , 2012, Pages 115-124

  Lorincz, András ^a   Palotai, Zsolt ^a,b   Szirtes, Gábor ^a  

^a EÖTVÖS LORÁND UNIVERSITY   (Hungary)

^b Sparsense Inc ^*  (United States)

Author keywords

1 Norm; Cross entropy method; Sparse coding

Indexed keywords

APPROXIMATE SOLUTION; CENTRALIZED ALGORITHMS; COMBINED METHOD; CROSS-ENTROPY METHOD; GREEDY METHOD; HARD PROBLEMS; ITERATIVE SOLUTIONS; NEURAL CIRCUITS; NEURAL INFORMATION PROCESSING; NON-ZERO COEFFICIENTS; OPTIMAL SOLUTIONS; OVER-COMPLETE; PRIORI KNOWLEDGE; PROBLEM SIZE; SPARSE CODING; SPARSE FORMS; SUBOPTIMAL SOLUTION;

ALGORITHMS; DATA PROCESSING; OPTIMAL SYSTEMS; OPTIMIZATION;

CODES (SYMBOLS);

ARTICLE; CLASSIFICATION ALGORITHM; CODING; COMPUTER; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; CORRELATION COEFFICIENT; ENTROPY; INFORMATION PROCESSING; MATHEMATICAL COMPUTING; NERVE CELL; NERVE CELL MEMBRANE POTENTIAL; PRIORITY JOURNAL; PROCESS MODEL; PROCESS OPTIMIZATION; SIGNAL TRANSDUCTION; STIMULUS RESPONSE; SYSTEM ANALYSIS;

EID: 83955161106     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2011.10.017     Document Type: Article

References (65)

1. Mairal J., Bach F., Ponce J., Sapiro G. Online learning for matrix factorization and sparse coding. J. Mach. Learn. Res. 2010, 11:19-60.
2. Schmidhuber J. Learning factorial codes by predictability minimization. Neural Comput. 1992, 4:863-879.
3. Lee A.B., Pedersen K.S., Mumford D.B. The nonlinear statistics of high-contrast patches in natural images. Int. J. Comput. Vision 2003, 54:83-103.
4. Olshausen B.A., Field D.J. Sparse coding with an overcomplete basis set: a strategy employed by V1?. Vision Res. 1997, 37:3311-3325.
5. Olshausen B.A., Field D.J. Sparse coding of sensory inputs. Curr. Opinion Neurobiol. 2004, 14:481-487.
6. Smith E.C., Lewicki M.S. Efficient auditory coding. Nature 2006, 439:978-982.
7. Barlow H.B. Possible principles underlying the transformation of sensory messages. Sensory Communication 1961, 217-234. MIT Press, Cambridge, MA. W.A. Rosenblith (Ed.).
8. Field D.J. What is the goal of sensory coding. Neural Comput. 1994, 6:559-601.
9. Mumford D. Neuronal architectures for pattern-theoretic problems. Large Scale Neuronal Theories of the Brain 1994, 125-152. MIT Press. C. Koch, J.L. Davis (Eds.).
10. Lee D.D., Seung H.S. Learning the parts of objects by non-negative matrix factorization. Nature 1999, 401:788-791.
11. Olshausen B.A. Sparse codes and spikes. Probabilistic Models of the Brain 2002, 257-272. MIT Press. R.P.N. Rao, B.A. Olshausen, M.S. Lewicki (Eds.).
12. Vinje W., Gallant J. Natural stimulation of the nonclassical receptive field increases information transmission efficiency in v1. J. Neurosci. 2002, 22:2904-2915.
13. Berkes P., White B., Fiser J. No evidence for active sparsification in the visual cortex. Advances in Neural Information Processing Systems 2009, vol. 22:108-116.
14. Graham D.J., Field D.J. Sparse coding in the neocortex. Evolution of Nervous Systems 2006, vol. III:181-187. Elsevier.
15. Natarajan B. Sparse approximate solutions to linear systems. SIAM J. Comput. 1995, 24:227-234.
16. Dai W., Milenkovic O. Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 2009, 55:2230-2249.
17. Rubinstein R., Kroese D.P. The Cross-Entropy Method 2004, Springer-Verlag, New York.
18. Lorincz A., Palotai Z., Szirtes G. Spike-based cross-entropy method for reconstruction. Neurocomputing 2008, 71:3635-3639.
19. Tibshirani R. Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B 1996, 58:267-288.
20. Chen S., Donoho D., Saunders M. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 2001, 43:129-159.
21. Donoho D.L., Elad M. Optimally sparse representation in general nonorthogonal dictionaries via l1 minimization. Proc. Natl. Acad. Sci. USA 2003, 100:2197-2202.
22. Donoho D.L. Compressed sensing. IEEE Trans. Inf. Theory 2006, 52:1289-1306.
23. Candès E., Romberg J., Tao T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 2006, 52:489-509.
24. T. Tao, Preprints in Sparse Recovery/Compressed Sensing, 2008 http://www.math.ucla.edu/~tao/preprints/sparse.html.
25. Boyd S., Vandenberghe L. Convex Optimization 2004, Cambridge University Press.
26. Kim S., Koh K., Lustig M., Boyd S., Gorinevsky D. An interior-point method for large-scale l1-regularized logistic regression. J. Mach. Learn. Res. 2007, 8:1519-1555.
27. Tropp J.A. Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 2004, 50:2231-2242.
28. Needell D., Vershynin R. Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math. 2009, 9:317-334.
29. Mallat S., Zhang Z. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 1993, 41:3397-3415.
30. B. Mailhe, R. Gribonval, F. Bimbot, P. Vandergheynst, A low complexity orthogonal matching pursuit for sparse signal approximation with shift-invariant dictionaries, in: ICASSP'09, IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3445-3448.
31. D. Donoho, Y. Tsaig, I. Drori, F. Starck, Sparse Solution of Underdetermined Linear Equations by Stagewise Orthogonal Matching Pursuit, Technical Report 2006-02, Stanford University, Stanford, CA, 2006.
32. Needell D., Tropp J.A. CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal. 2008, 26:301-321.
33. Rozell C.J., Johnson D.H., Baraniuk E.G., Olshausen B.A. Sparse coding via thresholding and local competition in neural circuits. Neural Comput. 2008, 20:2526-2563.
34. Lennie P. The cost of cortical computation. Curr. Biol. 2003, 13:493-497.
35. Willmore B., Tolhurst D. Characterising the sparseness of neural codes. Network: Comput. Neural Syst. 2001, 12:255-270.
36. Lee H., Battle A., Raina R., Ng A.Y. Efficient sparse coding algorithms. Advances in Neural Information Processing Systems, NIPS 2007, 801-808.
37. Y. Weiss, H. Chang, W. Freeman, Learning Compressed Sensing, 2007.
38. Hochreiter S., Schmidhuber J. Feature extraction through LOCOCODE. Neural Comput. 1999, 11:679-714.
39. Rubinstein R.Y. The cross-entropy method for combinatorial and continuous optimization. Methodol. Comput. Appl. Probab. 1999, 2:127-190.
40. Muehlenbein H. The equation for response to selection and its use for prediction. Evol. Comput. 1998, 5:303-346.
41. Allon G., Kroese D.P., Raviv T., Rubinstein R.Y. Application of the cross-entropy method to the buffer allocation problem in a simulation-based environment. Ann. Oper. Res. 2005, 134:137-151.
42. J. Keith, D.P. Kroese, Sequence alignment by rare event simulation, in: Proceedings of the 2002 Winter Simulation Conference, pp. 320-327.
43. Szita I., Lorincz A. Learning to play using low-complexity rule-based policies: illustrations through Ms. Pac-Man. J. Artif. Intell. Res. 2007, 30:659-684.
44. Szabó Z., Póczos B., Lorincz A. Cross-entropy optimization for independent process analysis. Independent Component Analysis and Blind Signal Separation, Lecture Notes in Computer Science 2006, vol. 3889:909-916. Springer.
45. de Boer P.-T., Kroese D.P., Mannor S., Rubinstein R.Y. A tutorial on the cross-entropy method. Ann. Oper. Res. 2004, 134:19-67.
46. I. Szita, A. Lorincz, Online Variants of the Cross-Entropy Method, 2008 http://arxiv.org/abs/0801.1988.
47. Carpenter G.A., Grossberg S. ART2: self-organization of stable category recognition codes for analog input patterns. Appl. Opt. 1987, 26:4919-4930.
48. Carpenter G., Grossberg S. A massively parallel architecture for a self-organizing neural pattern recognition machine. Comput. Vision Graphics Image Process. 1987, 37:54-115.
49. Riesenhuber M., Poggio T. Hierarchical models of object recognition in cortex. Nat. Neurosci. 1999, 2:1019-1025.
50. W. Candès, J. Romberg, l1-Magic: A Collection of Matlab Routines for Solving the Convex Optimization Programs Central to Compressive Sampling, online, 2006 http://www.acm.caltech.edu/l1magic/.
51. Olshausen B.A., Field D.J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381:607-609.
52. Rehn M., Sommer F.T. A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields. J. Comput. Neurosci. 2007, 22:135-146.
53. W.K. Coulter, C.J. Hillar, G. Isley, F.T. Sommer. Adaptive compressed sensing-a new class of self-organizing coding models for neuroscience, In: Proceedings of ICASSP'2010, pp. 5494-5497, doi:. http://10.1109/ICASSP.2010.5495209.
54. O'Reilly R., McClelland J. Hippocampal conjunctive encoding, storage, and recall: avoiding a tradeoff. Hippocampus 1994, 4:661-682.
55. Maass W. On the computational power of winner-take-all. Neural Comput. 2000, 12:2519-2535.
56. Pouget A., Dayan P., Zemel R. Information processing with population codes. Nat. Rev. Neurosci. 2000, 1:125-132.
57. Ma W.J., Beck J.M., Latham P.E., Pouget A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 2006, 9:1432-1438.
58. London M., Häusser M. Dendritic computation. Ann. Rev. Neurosci. 2005, 28:503-532.
59. Thorpe S.J. How can the human visual system process a natural scene in under 150ms? Experiments and neural network models. Eurorean Symposium on Artificial Neural Networks, D-Facto public 1997.
60. Rao R.P.N., Ballard D.H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 1999, 2:79-87.
61. Koch C., Poggio T. Predicting the visual world: silence is golden. Nat. Neurosci. 1999, 2:9-10.
62. Sillito A.M., Cudeiro J., Jones H.E. Always returning: feedback and sensory processing in visual cortex and thalamus. Trends Neurosci. 2006, 29:307-316.
63. Serre T., Oliva A., Poggio T. A feedforward architecture accounts for rapid categorization. Proc. Natl. Acad. Sci. 2007, 104:6424-6429.
64. Liu H., Agam Y., Madsen J.R., Kreiman G. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron 2009, 62:281-290.
65. Llinas R. I of the Vortex: From Neurons to Self 2002, The MIT Press.