Backpropagation-Decorrelation: Online recurrent learning with O(N) complexity

IEEE International Conference on Neural Networks - Conference Proceedings

Volumn 2, Issue , 2004, Pages 843-848

  Steil, Jochen J ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

NETWORK DYNAMICS; OPTIMIZATION PROBLEM; RECURRENT NETWORKS; TIME CONSTANTS;

ALGORITHMS; BACKPROPAGATION; ERROR ANALYSIS; LEARNING SYSTEMS; OPTIMIZATION; PROBLEM SOLVING; TIME SERIES ANALYSIS;

NEURAL NETWORKS;

EID: 10944225085     PISSN: 10987576     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IJCNN.2004.1380039     Document Type: Conference Paper

References (21)

1. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity," in Backpropagation: Theory, Architectures, and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Lawrence Erlbaum Publ., 1995, pp. 433-486.
2. B. A. Pearlmutter, "Gradient calculations for dynamic recurrent neural networks: A survey," IEEE Tansactions on Neural Networks, vol. 6, no. 5, pp. 1212-1228, 1995.
3. 3) time complexity learning algorithm for fully recurrent continually running networks," Neural Computation, vol. 4, no. 2, pp. 243-248, 1992.
4. R. J. Williams and J. Peng, "An efficient gradient-based algorithm for on-line training of recurrent network trajectories," Neural Computation, vol. 2, no. 4, pp. 490-501, 1990.
5. B. Hammer and J. J. Steil, "Tutorial: Perspectives on learning with recurrent neural networks," in Proc. of ESANN, 2002, pp. 357-368.
6. A. B. Atiya and A. G. Parlos, "New results on recurrent network training: Unifying the algorithms and accelerating convergence," IEEE Trans. Neural Networks, vol. 11, no. 9, pp. 697-709, 2000.
7. H. Jaeger, "Adaptive nonlinear system identification with echo state networks," in NIPS, 2002.
8. T. Natschläger, W. Maass, and H. Markram, "The "liquid computer": A novel strategy for real-time computing on time series," TELEMATIK, vol. 8, no. 1, pp. 39-43, 2002.
9. U. D. Schiller and J. J. Steil, "On the weight dynamcis of recurrent learning," in Proc. ESANN, 2003, pp. 73-78.
10. S. Choi, S. Amari, and A. Cichocki, "Natural gradient learning for spatio-temporal decorrelation: Recurrent network," IEICE Trans. Fundamentals, vol. E83-A, no. 12, 2000.
11. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. Wiley, 2001.
12. K. P. Körding and P. König, "A learning rule for dynamic recruitment and decorrelation," Neural Networks, vol. 13, pp. 1-9, 2000.
13. W. F. Chang and M. W. Mak, "A conjugate gradient learning algorithm for recurrent neural networks," Neurocomputing, vol. 24, no. 1-3, pp. 173-189, 1999.
14. E. P. dos Santos and F. J. V. Zuben, Recurrent Neural Networks: Design and Applications. CRC Press, 1999, ch. Efficient Second-Order Learning Algorithms for Discrete-Time Recurrent Neural Networks.
15. U. D. Schiller and J. J. Steil, "Analyzing the weight dynamics of recurrent learning algorithms," Neurocomputing, 2004, in press.
16. P. Tiňo, B. G. Home, and C. L. Giles, "Attractive periodic sets in descrete-time recurrent networks (with emphasis on fixed-point stability and bifurcations in two-neuron networks)," Neural Computation, vol. 13, pp. 1379-1414, 2001.
17. U. D. Schiller, "Analysis and comparison of algorithms for training recurrent neural networks," Master's thesis, Faculty of Technology, University of Bielefeld, 2003. [Online]. Available: http://www.ulfschiller.de/ publications/diploma.pdf
18. S. Hochreiter, "The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions," Int. J. Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 6, no. 2, pp. 107-116, 1998.
19. S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, vol. 9, pp. 1735-1780, 1997.
20. J. J. Steil and H. Ritter, "Recurrent learning of input-output stable behaviour in function space: A case study with the Roessler attractor," in Proc. ICANN 99. IEE, 1999, pp. 761-766.
21. J. J. Steil, "Local structural stability of recurrent networks with timevarying weights," Neurocomputing, vol. 48, no. 1-4, pp. 39-51, 2002.