Simple recurrent network trained by RTRL and extended Kalman filter algorithms

Neural Network World

Volumn 13, Issue 3, 2003, Pages 223-234

  Čerňanský, Michal ^a   Beňuškova, Lubica ^b  

^a SLOVAK UNIVERSITY OF TECHNOLOGY   (Slovakia)

^b Physics and Informatics   (Slovakia)

Author keywords

Extended Kalman filter; Next symbol prediction; Recurrent neural networks; Recursive languages

Indexed keywords

BACKPROPAGATION; COGNITIVE SYSTEMS; COMPUTATIONAL METHODS; GRADIENT METHODS; KALMAN FILTERING; LEARNING ALGORITHMS; NATURAL LANGUAGE PROCESSING SYSTEMS; REAL TIME SYSTEMS; RECURSIVE FUNCTIONS;

COGNITIVE SCIENCE; EXTENDED KALMAN FILTER; NEXT SYMBOL PREDICTION; REAL TIME RECURRENT LEARNING;

RECURRENT NEURAL NETWORKS;

EID: 0037624007     PISSN: 12100552     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (19)

1. Čerňanský M., Beňušková L.: Finite-state Reber automaton and the recurrent neural networks trained in supervised and unsupervised manner. In: H. Bischof, G. Dorffner and K. Hornik (eds), LNCS 2130. Artificial Neural Networks - ICANN'2001, Springer-Verlag, 2001, pp. 737-742.
2. Christiansen, M. H., Chater, N.: Toward a connectionist model of recursion in human linguistic performance. Cognitive Sci., 23, 1999, pp. 417-437.
3. Elman J. L.: Finding structure in time. Cognitive Sci., 14, 1990, pp. 179-211.
4. Hammer B., Tiño P.: Recurrent neural networks with small weights implement finite memory machines. To appear in Neural Computation.
5. Kolen, J. F.: The origin of clusters in recurrent neural network state space. In: Proc. 16th Annual Conf. of the Cognitive Sci. Soc., Hillsdale, NJ: Lawrence Erlbaum Associates, 1994, pp. 508-513.
6. Lawrence S., Giles C. L., Fong S.: Natural language grammatical inference with recurrent neural networks. IEEE Trans. Knowledge and Data Engineering, 12, 1, 2000, pp. 126-140.
7. Maybeck P. S.: Stochastic models, estimation, and control, 1, New York Academic Press, 1979.
8. Patel G. S.: Modeling nonlinear dynamics with extended Kalman filter trained recurrent multilayer perceptrons. Thesis, McMaster Univ., Canada, 2000.
9. Pérez-Ortiz J. A., Gers F. A., Eck D., Schmidhuber J.: Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets. Neural Networks, 16, 2, 2003, pp. 1-23.
10. Rodriguez P.: Simple recurrent networks learn contex-free and contex-sensitive languages by counting. Neural Computation, 13, 2001, pp. 2093-2118.
11. Ron D., Singer Y., Tishby N.: The power of amnesia. Machine Learning, 25, 1996.
12. Tiño P., Hammer B.: Architectural bias in recurrent neural networks - fractal analysis. To appear in Neural Computation.
13. Tiño P., Čerňanský M., Beňušková L.: Markovian architectural bias of recurrent neural networks. To appear in IEEE Trans. Neural Net.
14. Welch G., Bishop G.: An introduction to the Kalman filter. TR95-041, Dept. Computer Science, Univ. North Carolina, 1995.
15. Werbos P. J.: Backpropagation through time; what it does and how to do it. Proceedings of the IEEE, 78, 1990, pp. 1550-1560.
16. Williams R. J.: Some observations on the use of the extended Kalman filter as a recurrent network learning algorithm. TR NU-CCS-92-1, Boston, 1992.
17. Williams R. J.: Training recurrent networks using the extended Kalman filter. In: Proc. Intl. Joint Conf. Neural Networks, 4, Baltimore, June 1992, pp. 241-246.
18. Williams R. J., Zipser D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1, 1989 pp. 270-280.
19. Williams R. J., Zipser D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. In: Y. Chauvin and D. E. Rumelhãrt (eds). Backpropagation: Theory, Architectures and Applications, Lawrence Erlbaum Publishers, Hillsdale, N. J., 1995, pp. 433-486.