Learning to forget: Continual prediction with LSTM

Neural Computation

Volumn 12, Issue 10, 2000, Pages 2451-2471

  Gers, Felix A ^a   Schmidhuber, Jürgen ^a   Cummins, Fred ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; NONLINEAR SYSTEM; SHORT TERM MEMORY;

ALGORITHMS; MEMORY, SHORT-TERM; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS;

EID: 0034293152     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/089976600300015015     Document Type: Article

References (22)

1. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
2. Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989). Finite-state automata and simple recurrent networks. Neural Computation, 1, 372-381.
3. Cummins, F., Gers, F., & Schmidhuber, J. (1999). Language identification from prosody without explicit features. In Proceedings of EUROSPEECH'99 (Vol. 1, pp. 371-374).
4. Darken, C. (1995). Stochastic approximation and neural network learning. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 941-944). Cambridge, MA: MIT Press.
5. Doya, K., & Yoshizawa, S. (1989). Adaptive neural oscillator using continuous-time backpropagation learning. Neural Networks, 2(5), 375-385.
6. Fahlman, S. E. (1991). The recurrent cascade-correlation learning algorithm. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems, 3 (pp. 190-196). San Mateo, CA: Morgan Kaufmann.
7. Gers, F. A., Schmidhuber, J., & Cummins, F. (1999). Learning to forget: Continual prediction with LSTM (Tech. Rep. No. IDSIA-01-99). Lugano, Switzerland: IDSIA.
8. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Technische Universität München. Available online at www7. informatik.tu-muenchen.de/~hochreit.
9. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780.
10. Jordan, M. I. (1986). Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of the Eighth Annual Cognitive Science Society Conference. Hillsdale, NJ: Erlbaum.
11. Lin, T., Horne, B. G., Tiño, P., & Giles, C. L. (1996). Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7(6), 1329-1338.
12. Mozer, M. C. (1989). A focused backpropagation algorithm for temporal pattern processing. Complex Systems, 3, 349-381.
13. Pearlmutter, B. A. (1995). Gradient calculation for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5), 1212-1228.
14. Robinson, A. J., & Fallside, F. (1987). The utility driven dynamic error propagation network. (Tech. Rep. No. CUED/F-INFENG/TR.1). Cambridge: Cambridge University Engineering Department.
15. Schmidhuber, J. (1989). The neural bucket brigade: A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 1(4), 403-412.
16. 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2), 243-248.
17. Schraudolph, N. (1999). A fast, compact approximation of the exponential function. Neural Computation, 11(4), 853-862.
18. Smith, A. W., & Zipser, D. (1989). Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems, 1(2), 125-131.
19. Waibel, A. (1989). Modular construction of time-delay neural networks for speech recognition. Neural Computation, 1(1), 39-46.
20. Werbos, P. J. (1988). Generalisation of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 339-356.
21. Williams, R. J., & Peng, J. (1990). An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 2(4), 490-501.
22. Williams, R. J., & Zipser, D. (1992). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures and applications. Hillsdale, NJ: Erlbaum.