Initialization and self-organized optimization of recurrent neural network connectivity

HFSP Journal

Volumn 3, Issue 5, 2009, Pages 340-349

  Boedecker, Joschka ^a   Obst, Oliver ^b   Michael Mayer, N ^c,d   Asada, Minoru ^a,c  

^a OSAKA UNIVERSITY   (Japan)

^b CSIRO   (Australia)

^c JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

^d NATIONAL CHUNG CHENG UNIVERSITY   (Taiwan)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 78349272309     PISSN: 19552068     EISSN: 1955205X     Source Type: Journal    
DOI: 10.2976/1.3240502     Document Type: Article

References (38)

1. Bengio, Y, Simard, P, and Frasconi, P (1994). "Learning long-term dependencies with gradient descent is difficult." IEEE Trans. Neural Netw. 5(2), 157-166.
2. Bertschinger, N, and Natschläger, T (2004). "Real-time computation at the edge of chaos in recurrent neural networks." Neural Comput. 16(7), 1413-1436.
3. Daoudal, G, and Debanne, D (2003). "Long-term plasticity of intrinsic excitability: learning rules and mechanisms." Learn. Memory 10, 456-465.
4. Douglas, R, Markram, H, and Martin, K (2004). "Neocortex." The Synaptic Organization of the Brain, 5th Ed., Shepard, GM, ed., pp. 499-558, Oxford University Press, NewYork.
5. Doya, K (1995). "Recurrent networks: supervised learning." The Handbook of Brain Theory and Neural Networks, Arbib, MA, ed., pp. 796-800, MIT Press, Cambridge, MA.
6. Field, DJ (1994). "What is the goal of sensory coding?" Neural Comput. 6(4), 559-601.
7. Hajnal, M, and Lorincz, A (2006). "Critical echo state networks." Artificial Neural Networks-ICANN 2006, 658-667.
8. Hammer, B, Micheli, A, Neubauer, N, Sperduti, A, and Strickert, M (2005). "Self organizing maps for time series." Proceedings of WSOM 2005, 115-122.
9. Hammer, B, Schrauwen, B, and Steil, JJ (2009). "Recent advances in efficient learning of recurrent networks." Proceedings of the European Symposium on Artificial Neural Networks (ESANN), Verleysen, M, ed., 213-226.
10. Hochreiter, S (1991). "Untersuchungen zu dynamischen neuronalen netzen." Diploma, Technische Universität München.
11. Hochreiter, S, Bengio, Y, Frasconi, P, and Schmidhuber, J (2001). "Gradient flow in recurrent nets: the difficulty of learning long-term dependencies." A Field Guide to Dynamical Recurrent Neural Networks, Kremer, SC, and Kolen, JF, eds., IEEE, NewYork.
12. Hochreiter, S, and Schmidhuber, J (1997a). "Flat minima." Neural Comput. 9(1), 1-42.
13. Hochreiter, S, and Schmidhuber, J (1997b). "Long short-term memory." Neural Comput. 9(8), 1735-1780.
14. Jaeger, H (2001a). "The "echo state" approach to analysing and training recurrent neural networks." GMD-German National Research Institute for Computer Science, Technical Report No. 148.
15. Jaeger, H (2001b). "Short term memory in echo state networks." GMD- German National Research Institute for Computer Science, Technical Report No. 152.
16. Jaeger, H (2003). "Adaptive nonlinear system identification with echo state networks." Advances in Neural Information Processing Systems 15 (NIPS 2002), Becker, S, Thrun, S, and Obermayer, K, eds., pp. 593-600, MIT Press, Cambridge, MA.
17. Jaeger, H, and Haas, H (2004). "Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication." Science 304(5667), 78-80.
18. Klapper-Rybicka, M, Schraudolph, NN, and Schmidhuber, J (2001). "Unsupervised learning in lstm recurrent neural networks." ICANN-Lecture Notes in Computer Science, Vol. 2130, G. Dorffner, H. Bischof, and K. Hornik, eds., pp. 684-691, Springer, New York.
19. Lazar, A, Pipa, G, and Triesch, J (2007). "Fading memory and time series prediction in recurrent networks with different forms of plasticity." Neural Networks 20(3), 312-322.
20. Lukosevicius, M, and Jaeger, H (2007). "Overview of reservoir recipes." Jacobs University, Technical Report No. 11, School of Engineering and Science.
21. Maass, W, and Markram, H (2006). "Theory of the computational function of microcircuit dynamics." Microcircuits. The Interface Between Neurons and Global Brain Function, Grillner, S, and Graybiel, AM, eds., pp. 371-392, MIT Press, Cambridge, MA.
22. Maass, W, Natschläger, T, and Markram, H (2002). "Real-time computing without stable states: a new framework for neural computation based on perturbations." Neural Comput. 14(11), 2531-2560.
23. Ozturk, MC, Xu, D, and Príncipe, JC (2007). "Analysis and design of echo state networks." Neural Comput. 19(1), 111-138.
24. Rad, AA, Jalili, M, and Hasler, M (2008). "Reservoir optimization in recurrent neural networks using Kronecker kernels." Proceedings of the IEEE Int. Symposium on Circuits and Systems, 868-871.
25. Schmidhuber, J, Wierstra, D, Gagliolo, M, and Gomez, F (2007). "Training recurrent networks by Evolino." Neural Comput. 19(3), 757-779.
26. Schmidhuber, J, Wierstra, D, and Gomez, FJ (2005). "Evolino: hybrid neuroevolution/optimal linear search for sequence learning." Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, L.P. Kaelbling, and A. Saffiotti, eds., pp. 853-858.
27. Schrauwen, B, Wardermann, M, Verstraeten, D, Steil, JJ, and Stroobandt, D (2008). "Improving reservoirs using intrinsic plasticity." Neurocomputing 71(7-9), 1159-1171.
28. Silberberg, G, Gupta, A, and Markram, H (2002). "Stereotypy in neocortical microcircuits." Trends Neurosci. 25(5), 227-230.
29. Steil, JJ (2004). "Backpropagation-decorrelation: online recurrent learning with o(n) complexity." Proceedings of the International Joint Conference on Neural Networks (IJCNN), Vol. 1, 843-848.
30. Steil, JJ (2007). "Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning." Neural Networks 20(3), 353-364.
31. Triesch, J (2005). "A gradient rule for the plasticity of a neuron's intrinsic excitability." Artificial Neural Networks: Biological Inspirations-ICANN 2005, pp. 65-70, Springer, NewYork.
32. Triesch, J (2007). "Synergies between intrinsic and synaptic plasticity mechanisms." Neural Comput. 19(4), 885-909.
33. Verstraeten, D, Schrauwen, B, D'Haene, M, and Stroobandt, D (2007). "An experimental unification of reservoir computing methods." Neural Networks 20(3), 391-403.
34. Werbos, PJ (1990). "Backpropagation through time: what it does and how to do it." Proc. IEEE 78(10), 1550-1560.
35. White, OL, Lee, DD, and Sompolinsky, H (2004). "Short-term memory in orthogonal neural networks." Phys. Rev. Lett., 92(14), 148102.
36. Wierstra, D, Gomez, FJ, and Schmidhuber, J (2005). "Modeling systems with internal state using Evolino." GECCO '05: Proceedings of the 2005 Conference on Genetic and Evolutionary Computation, pp. 1795-1802, ACM, NewYork.
37. Williams, RJ, and Zipser, D (1989). "A learning algorithm for continually running fully recurrent neural networks." Neural Comput. 1(2), 270-280.
38. Zhang, W, and Linden, DJ (2003). "The other side of the engram: experience-driven changes in the neuronal intrinsic excitability." Nat. Rev. Neurosci. 4, 885-900.