Short-term memory capacity in networks via the restricted isometry property

Neural Computation

Volumn 26, Issue 6, 2014, Pages 1198-1235

  Charles, Adam S ^a   Yap, Han Lun ^a   Rozell, Christopher J ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; BRAIN CORTEX; HUMAN; NERVE CELL NETWORK; NONLINEAR SYSTEM; PHYSIOLOGY; SHORT TERM MEMORY;

CEREBRAL CORTEX; HUMANS; MEMORY, SHORT-TERM; MODELS, NEUROLOGICAL; NERVE NET; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS;

EID: 84899857287     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00590     Document Type: Letter

References (45)

1. Bajwa, W. U., Sayeed, A. M., & Nowak, R. (2009). A restricted isometry property for structurally-subsampled unitary matrices. In Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing (pp. 1005-1012). Piscataway, NJ: IEEE.
2. Balavoine, A., Romberg, J., & Rozell, C. J. (2012). Convergence and rate analysis of neural networks for sparse approximation. IEEE Transactions on Neural Networks and Learning Systems, 23, 1377-1389.
3. Balavoine, A., Rozell, C. J., & Romberg, J. (2013). Convergence speed of a dynamical system for sparse recovery. IEEE Transactions on Signal Processing, 61, 4259-4269.
4. Baum, E. B., Moody, J., & Wilczek, F. (1988). Internal representations for associative memory. Biological Cybernetics, 92, 217-228.
5. Becker, S., Candes, E. J.,&Grant,M. (2011). Templates for convex cone problemswith applications to sparse signal recovery. Mathematical Programming Computation, 3.
6. Buonomano, D. V., & Maass, W. (2009). State-dependent computations: Spatiotemporal processing in cortical networks. Nature Reviews Neuroscience, 10, 113-125.
7. Büsing, L., Schrauwen, B., & Legenstein, R. (2010). Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Computation, 22, 1272-1311.
8. Candes, E. J. (2006). Compressive sampling. In Proc. Int. Congr.Mathematicians (vol. 3, pp. 1433-1452). Zurich: European Mathematical Society Publishing House.
9. Candes, E. J., Romberg, J.,&Tao, T. (2006).Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52, 489-509.
10. Candes, E. J.,&Tao, T. (2006).Near-optimal signal recovery fromrandom projections: Universal encoding strategies? IEEE Transactions on Information Theory, 52, 5406-5425.
11. Davenport, M. A., Boufounos, P. T., Wakin, M. B., & Baraniuk, R. G. (2010). Signal processing with compressive measurements. IEEE J. Sel. Topics Signal Process., 4, 445-460.
12. De La Peña, V. H., & Gińe, E. (1999). Decoupling: From dependence to independence. New York: Springer-Verlag.
13. Diaconis, P., & Shahshahani, M. (1994). On the eigenvalues of random matrices. Journal of Applied Probability, 31, 49-62.
14. Donoho, D., & Tanner, J. (2005). Sparse nonnegative solution of underdetermined linear equations by linear programming. Proceedings of the National Academy of Sciences of the United States of America, 102, 9446-9451.
15. Eftekhari,A.,Yap, H. L.,Rozell, C. J.,&Wakin, M. B. (in press).The restricted isometry property for random block diagonal matrices. Applied and ComputationalHarmonic Analysis.
16. Elad, M., Figueiredo, M., & Ma, Y. (2008). On the role of sparse and redundant representations in image processing. In Proceedings of the IEEE, 98, 972-982.
17. Ganguli, S.,Huh,D.,&Sompolinsky, H. (2008).Memory traces in dynamical systems. Proceedings of the National Academy of Sciences, 105, 18970-18975.
18. Ganguli, S., & Sompolinsky, H. (2010). Short-term memory in neuronal networks through dynamical compressed sensing. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, & A. Culotta (Eds.), Neural information processing systems, 23. Red Hook, NY: Curran.
19. Ganguli, S., & Sompolinsky, H. (2012). Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. Annual Review of Neuroscience, 35, 485-508.
20. Haupt, J., Bajwa, W. U., Raz, G., & Nowak, R. (2010). Toeplitz compressed sensing matrices with applications to sparse channel estimation. IEEE Transactions on Information Theory, 56, 5862-5875.
21. Hermans, M., & Schrauwen, B. (2010). Memory in linear recurrent neural networks in continuous time. Neural Networks, 23, 341-355.
22. Hu, T., Genkin, A., & Chklovskii, D. B. (2012). A network of spiking neurons for computing sparse representations in an energy-efficient way. Neural Computation, 24, 2852-2872.
23. Isley, G., Hillar, C. J.,&Sommer, F. T. (2011).Deciphering subsampled data: Adaptive compressive sampling as a principle of brain communication. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, & A. Culotta (Eds.), Advances in neural information processing systems, 23. Red Hook, NY: Curran.
24. Jaeger, H. (2001). Short term memory in echo state networks (GMD Report 152). St. Augustin: German National Research Center for Information Technology.
25. Jaeger, H., & Haas, H. (2004). Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science, 304, 78-80.
26. Krahmer, F., Mendelson, S., & Rauhut, H. (2012). Suprema of chaos processes and the restricted isometry property. arXiv preprint arXiv:1207.0235.
27. Ledoux, M., & Talagrand, M. (1991). Probability in Banach spaces: Isoperimetry and processes. New York: Springer.
28. Legenstein, R., & Maass, W. (2007). Edge of chaos and prediction of computational performance for neural circuit models. Neural Networks, 20, 323- 334.
29. Maass, W., Natschl̈ager, T., & Markram, H. (2002). Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14, 2531-2560.
30. Mayor, J.,&Gerstner,W. (2005). Signal buffering in random networks of spiking neurons: Microscopic versus macroscopic phenomena. Physical Review E, 72, 051906.
31. Mongillo, G., Barak, O., & Tsodyks, M. (2008). Synaptic theory of working memory. Science, 319, 1543-1546.
32. Olshausen, B. A., & Field, D. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
33. Park, J. Y., Yap, H. L., Rozell, C. J., & Wakin, M. B. (2011). Concentration of measure for block diagonal matrices with applications to compressive signal processing. IEEE Transactions on Signal Processing, 59, 5859-5875.
34. Rauhut,H. (2010). Compressive sensing and structured randommatrices. Theoretical Found. and Numerical Methods for Sparse Recovery, 9, 1-92.
35. Rhen, M., & Sommer, F. T. (2007). A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields. Journal of Computational Neuroscience, 22, 135-146.
36. Rozell, C. J., Johnson,D.H., Baraniuk, R.G.,&Olshausen, B. A. (2010). Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20, 2526-2563.
37. Rudelson, M., & Vershynin, R. (2008). On sparse reconstruction from Fourier and gaussian measurements. Comms. Pure and Applied Math., 61, 1025-1045.
38. Shapero, S., Charles, A. S., Rozell, C., & Hasler, P. (2011). Low power sparse approximation on reconfigurable analog hardware. IEEE J. Emer. Sel. Top. in Circ. and Sys., 2, 530-541.
39. Strauss, T., Wustlich, W., & Labahn, R. (2012). Design strategies for weight matrices of echo state networks. Neural Computation, 24, 3246-3276.
40. Tropp, J. A., Laska, J. N., Duarte, M. F., Romberg, J. K., & Baraniuk, R. G. (2009). Beyond Nyquist: Efficient sampling of sparse bandlimited signals. IEEE Trans. Inform. Theory, 56, 520-540.
41. Vapnik, V. N., & Chervonenkis, A. Y. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16, 264-280.
42. Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Y. Eldar & G. Kutyniok (Eds.) Compressed sensing, theory and applications (pp. 210-260). Cambridge: Cambridge University Press.
43. Wallace, E., Hamid, R. M., & Latham, P. E. (2013). Randomly connected networks have short temporal memory. Neural Computation, 25, 1408-1439.
44. White, O. L., Lee, D. D.,&Sompolinsky, H. (2004). Short-termmemory in orthogonal neural networks. Physical Review Lett., 92, 148102.
45. Zhu, M., & Rozell, C. (2013). Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system. PLoS Computational Biology, 9, e1003191.