Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets

Neural Networks

Volumn 16, Issue 2, 2003, Pages 241-250

  Pérez Ortiz, Juan Antonio ^a   Gers, Felix A ^b   Eck, Douglas ^c   Schmidhuber, Jürgen U ^c  

^a UNIVERSITY OF ALICANTE   (Spain)

^b Mantik Bioinformatik GmbH   (Germany)

^c DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

Context sensitive language inference; Decoupled extended Kalman filter; Long short term memory; Online prediction; Recurrent neural networks

Indexed keywords

ALGORITHMIC LANGUAGES; KALMAN FILTERING; PROBLEM SOLVING; ROBUSTNESS (CONTROL SYSTEMS);

LONG SHORT-TERM MEMORY (LSTM);

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; FILTER; LONG TERM MEMORY; PRIORITY JOURNAL; PROBLEM SOLVING; SHORT TERM MEMORY; TECHNIQUE;

ALGORITHMS; MEMORY, SHORT-TERM; NEURAL NETWORKS (COMPUTER);

EID: 0038764011     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(02)00219-8     Document Type: Article

References (27)

1. Bakker B. Reinforcement learning with long short-term memory. Dietterich T.G., Becker S., Ghahramani Z. Advances in neural information processing systems. 2001;MIT Press, Cambridge, MA.
2. Boden M., Wiles J. Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science. 12:(3):2000.
3. n language. Proceedings of the 6th Conference on Neural Information Processing. 1999;. pp. 508-513.
4. Elman J.L. Finding structure in time. Cognitive Science. (14):1990;179-211.
5. Feldkamp L.A., Puskorius G.V. Training controllers for robustness: multi-stream DEKF. IEEE International Conference on Neural Networks. 1994;2377-2382.
6. Gers F.A., Schmidhuber J. LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks. 12:(6):2002;1333-1340.
7. Gers F.A., Schmidhuber J., Cummins F. Learning to forget: continual prediction with LSTM. Proceedings of the International Conference on Artificial Neural Networks. 1999;850-855.
8. Gers F.A., Schmidhuber J., Cummins F. Learning to forget: continual prediction with LSTM. Neural Computation. 12:(10):2000;2451-2471.
9. Haykin S. Neural networks: a comprehensive foundation. 2nd ed. 1999;Prentice-Hall, New Jersey.
10. Haykin S. Kalman filtering and neural networks. 2001;Wiley, New York.
11. Hochreiter S., Bengio Y., Frasconi P., Schmidhuber J. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. Kremer S.C., Kolen J.F. A field guide to dynamical recurrent neural networks. 2001;IEEE Press, New York.
12. Hochreiter S., Schmidhuber J. Long short-term memory. Neural Computation. 9:(8):1997;1735-1780.
13. Hochreiter S., Younger A.S., Conwell P.R. Learning to learn using gradient descent. Proceedings of the International Conference on Artificial Neural Networks. Proceedings of the International Conference on Artificial Neural Networks. Vol. 2130:2001;Springer, Berlin. pp. 87-94.
14. Pearlmutter B.A. Gradient calculations for dynamic recurrent neural networks: a survey. IEEE Transactions on Neural Networks. 6:(5):1995;1212-1228.
15. Plaut, D. C., Nowlan, S. J., & Hinton, G. E. (1986). Experiments on learning back propagation. Technical Report CMU-CS-86-126, Carnegie-Mellon University, Pittsburgh, PA.
16. Puskorius G.V., Feldkamp L.A. Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks. 5:(2):1994;279-297.
17. Robinson A.J., Fallside F. A recurrent error propagation speech recognition system. Computer Speech and Language. 5:1991;259-274.
18. Rodriguez P., Wiles J. Advances in Neural Information Processing Systems, 10. Advances in Neural Information Processing Systems, 10. 1998;MIT Press, Cambridge, MA. pp. 87-93.
19. Rodriguez P., Wiles J., Elman J. A recurrent neural network that learns to count. Connection Science. 11:(1):1999;5-40.
20. Schmidhuber J. A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science. 1:(4):1989;403-412.
21. 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation. 4:(2):1992;243-248.
22. Smith A.W., Zipser D. Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems. 1:(2):1989;125-131.
23. Sun, G. Z., Giles, C. L., Chen, H. H., & Lee, Y. C. (1993). The neural network pushdown automaton: model, stack and learning simulations. Technical Report CS-TR-3118, University of Maryland, College Park.
24. Wiles J., Elman J. Learning to count without a counter: a case study of dynamics and activation landscapes in recurrent networks. Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society. 1995;MIT Press, Cambridge, MA. pp. 482-487.
25. Williams R.J., Peng J. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation. 2:(4):1990;490-501.
26. Williams R.J., Zipser D. A learning algorithm for continually training recurrent neural networks. Neural Computation. 1:1989;270-280.
27. Williams R.J., Zipser D. Gradient-based learning algorithms for recurrent networks and their computational complexity. Chauvin Y., Rumelhart D.E. Back-propagation: theory, architectures and applications. 1992;Erlbaum, Hillsdale, NJ.