Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons

Neural Computation

Volumn 22, Issue 5, 2010, Pages 1272-1311

  Büsing, Lars ^a   Schrauwen, Benjamin ^b   Legenstein, Robert ^a  

^a GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

^b GHENT UNIVERSITY   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; MEMORY; NERVE CELL; TIME;

ALGORITHMS; ANIMALS; COMPUTER SIMULATION; MEMORY; NEURAL NETWORKS (COMPUTER); NEURONS; TIME FACTORS;

EID: 77953355233     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2009.01-09-947     Document Type: Article

References (48)

1. Athreya, K.B., & Ney, P.E. (1972). Branching processes. Berlin: Springer.
2. Beggs, J.M., & Plenz, D. (2003). Neuronal avalanches in neocortical circuits. Journal of Neuroscience, 23(35), 11167-11177.
3. Bertschinger, N., &Natschl̈ager, T. (2004). Real-time computation at the edge of chaos in recurrent neural networks. Neural Computation, 16(7), 1413-1436.
4. DeFelipe, J., & Fariñnas, I. (1992). The pyramidal neuron of the cerebral cortex: Morphological and chemical characteristics of the synaptic inputs. Prog. Neurobiol., 39(6), 563-607.
5. Derrida, B., & Pomeau, Y. (1986). Random networks of automata: A simple annealed approximation. Europhysics Letters, 1(2), 45-49.
6. Derrida, B., & Stauffer, D. (1986). Phase transitions in two-dimensional Kauffman cellular automata. Europhys. Lett., 2, 739-745.
7. Destexhe, A., Rudolph, M., & Paré, D. (2003). The high-conductance state of neocortical neurons in vivo. Nat. Rev. Neurosci., 4(9), 739-751.
8. Ganguli, S., Huh, D., &Sompolinsky, H. (2008).Memory traces in dynamical systems. PNAS, 15(48), 18970-18975.
9. Häusler, S., & Maass, W. (2007). A statistical analysis of information processing properties of lamina-specific cortical microcircuit models. Cerebral Cortex, 17(1), 149-162.
10. Jaeger, H. (2001). The " echo state" approach to analyzing and training recurrent neural networks (GMD Rep. 148). St. Augustin: German National Research Center for Information Technology.
11. Jaeger, H. (2002). Short term memory in echo state networks (GMD Rep. 152). St. Augustin: German National Research Center for Information Technology.
12. Jaeger, H. (2007). Echo state networks. Scholarpedia, 2(9), 2330.
13. Jaeger, H., & Haas, H. (2004). Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science, 304, 78-80.
14. Jaeger, H., Lukoševičius, M., Popovici, D., &Siewert, U. (2007).Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks, 20(3), 335-352.
15. Joshi, P., & Maass, W. (2005).Movement generation with circuits of spiking neurons. Neural Computation, 17(8), 1715-1738.
16. Joshi, P., & Triesch, J. (2008). A globally asymptotically stable plasticity rule for firing rate homeostasis. In Proceedings of the International Conference on Neural Networks (ICANN). Berlin: Springer.
17. Katok, A., & Hasselblatt, B. (1995). Introduction to the modern theory of dynamical systems. Cambridge: Cambridge University Press.
18. Kauffman, S.A. (1969). Metabolic stability and epigenesis in randomly connected nets. J. Theoret. Biol., 22, 437-467.
19. Langton, C.G. (1990). Computation at the edge of chaos. Physica D, 42, 12-37.
20. Laughlin, S., de Ruyter van Steveninck, R., & Anderson, J. (1998). The metabolic cost of neural information. Nature Neuroscience, 1, 36-41.
21. Lazar, A., Pippa, G., & Triesch, J. (2007). Fading memory and time series prediction in recurrent networks with different forms of plasticity. Neural Networks, 20, 312-322.
22. Legenstein, R., & Maass, W. (2007a). Edge of chaos and prediction of computational performance for neural microcircuit models. Neural Networks, 20, 323-334.
23. Legenstein, R., & Maass, W. (2007b). What makes a dynamical system computationally powerful? In S. Haykin, J.C. Principe, T. Sejnowski, & J. McWhirter (Eds.), New directions in statistical signal processing: From systems to brain (pp. 127-154). Cambridge, MA: MIT Press.
24. Legenstein, R., Markram, H., &Maass, W. (2003). Input prediction and autonomous movement analysis in recurrent circuits of spiking neurons. Reviews in the Neurosciences, 14(1-2), 5-19.
25. Lennie, P. (2003). The cost of cortical computation. Current Biology, 13, 493-497.
26. Levy, W.B., & Baxter, R.A. (1996). Energy efficient neural codes. Neural Computation, 8(3), 531-543.
27. Lukoševičius, M., & Jaeger, H. (2007). Overview of reservoir recipes (Tech. Rep. 11). Bremen: Jacobs University Bremen.
28. Luque, B., & Sol'e, R. V. (2000). Lyapunov exponents in random Boolean networks. Physica A, 284, 33-45.
29. Maass, W., Natschläger, T., & Markram, H. (2002). Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11), 2531-2560.
30. Mayor, J., &Gerstner, W. (2005). Signal buffering in random networks of spiking neurons: Microscopic versus macroscopic phenomena. Physical Review E, 72, 051906.
31. Mitchell, M., Hraber, P.T., & Crutchfield, J.P. (1993). Revisiting the edge of chaos: Evolving cellular automata to perform computations. Complex Systems, 7, 89-130.
32. Natschl̈ager, T., Bertschinger, N., & Legenstein, R. (2005). At the edge of chaos: Realtime computations and self-organized criticality in recurrent neural networks. In L.K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems, 17 (pp. 145-152). Cambridge, MA: MIT Press.
33. Packard, N. (1988). Adaption towards the edge of chaos. In J.A.S. Kelso, A.J. Mandell, & M.F. Shlesinger (Eds.), Dynamic patterns in complex systems (pp. 293-301). Singapore:World Scientific.
34. Schrauwen, B., Wardermann, M., Verstraeten, D., Steil, J.J., & Stroobandt, D. (2008). Improving reservoirs using intrinsic plasticity. Neurocomputing, 71, 1159-1171.
35. Shmulevich, I., Dougherty, E., Kim, S., & Zhang, W. (2002). Probabilistic Boolean networks: A rule-based uncertainty model for gene regulatory networks. Bioinformatics, 18(2), 261-274.
36. Solé, R., Luque, B., & Kauffman, S. (2000). Phase transitions in random networks with multiple states (Tech. Rep. 00-02-011). Santa Fe: Santa Fe Institute.
37. Steil, J.J. (2007). Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning. Neural Networks, 20(3), 353-364.
38. Triesch, J. (2007). Synergies between intrinsic and synaptic plasticity mechanisms. Neural Compuatation, 19(4), 885-909.
39. Tsodyks, M., &Markram, H. (1997). The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proc. Nat. Acad. Sci. USA, 94, 719-723.
40. Vandoorne, K., Dierckx, W., Schrauwen, B., Verstraeten, D., Baets, R., Bienstman, P., et al. (2008). Toward optical signal processing using photonic reservoir computing. Optics Express, 16(15), 11182-11192.
41. Vapnik, V.N. (1998). Statistical learning theory. New York:Wiley.
42. Verstraeten, D., Schrauwen, B., D'Haene, M., & Stroobandt, D. (2007). A unifying comparison of reservoir computing methods. Neural Networks, 20, 391-403.
43. Verstraeten, D., Schrauwen, B., Stroobandt, D., & Campenhout, J.V. (2005). Isolated word recognition with the liquid state machine: A case study. Information Processing Letters, 95(6), 521-528.
44. Verstraeten, D., Xavier-de Souza, S., Schrauwen, B., Suykens, J., Stroobandt, D., & Vandewalle, J. (2008). Pattern classification with cnns as reservoirs. In Proceedings of the International Symposium on Nonlinear Theory and Its Applications (NOLTA). Tokyo: Institute of Electronics, Information and Communication Engineers.
45. Vogels, T.P., Rajan, K., & Abbott, L. (2005). Neural networks dynamics. Annual Review of Neuroscience, 28, 357-376.
46. White, O.L., Lee, D.D., &Sompolinsky, H. (2004). Short-termmemory in orthogonal neural networks. Phys. Rev. Letters, 92(14), 148102.
47. Wolfram, S. (1984). Universality and complexity in cellular automata. Physica D, 10, 1-35.
48. Zinn-Justin, J. (2003). Phase transitions and renormalization group: From theory to numbers. New York: Oxford University Press.