Learning precise timing with LSTM recurrent networks

Journal of Machine Learning Research

Volumn 3, Issue 1, 2003, Pages 115-143

  Gers, Felix A ^a   Schraudolph, Nicol N ^b   Schmidhuber, Jürgen ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

^b ETH ZURICH   (Switzerland)

Author keywords

Long Short Term Memory; Recurrent Neural Networks; Timing

Indexed keywords

MARKOV PROCESSES; RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION; VIRTUAL REALITY;

HIDDEN MARKOV MODELS (HMM);

LEARNING SYSTEMS;

EID: 0041965934     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/153244303768966139     Document Type: Article

References (24)

1. Y. Bengio and P. Frasconi. An input output HMM architecture. In Advances in Neural Information Processing Systems 7, San Mateo CA, 1995. Morgan Kaufmann.
2. Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994.
3. A.B. Bulsari and H. Saxén. A recurrent network for modelling noisy temporal sequences. Neuro-computing, 7:29-40, 1995.
4. Fred Cummins, Felix Gers, and Jürgen Schmidhuber. Language identification from prosody without explicit features. In Proceedings of EUROSPEECH'99, volume 1, pages 371-374, 1999.
5. K. Doya and S. Yoshizawa. Adaptive neural oscillator using continuous-time backpropagation learning. Neural Networks, 2(5):375-385, 1989.
6. D. Eck and J. Schmidhuber. Learning the long-term structure of the blues. In Proc. Intl. Conf. Artificial Neural Networks, Lecture Notes in Computer Science. Springer Verlag, Berlin (in press), 2002.
7. Mikel L. Forcada and Rafael C. Carrasco. Learning the initial state of a second-order recurrent neural network during regular-language inference. Neural Computation, 7(5):923-930, 1995.
8. F. A. Gers and J. Schmidhuber. LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks, 2001.
9. F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10):2451-2471, 2000.
10. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991. See www7.informatik.tu-muenchen.de/̃hochreit.
11. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
12. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.
13. Y. LeCun, L. Bottou, G.B. Orr, and K.-R. Müller. Efficient backprop. In Genevieve B. Orr and Klaus-Robert Müller, editors, Neural Networks - Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pages 5-50. Springer Verlag, Berlin, 1998.
14. B. A. Pearlmutter. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5):1212-1228, 1995.
15. D. C. Plaut, S. J. Nowlan, and G. E. Hinton. Experiments on learning back propagation. Technical Report CMU-CS-86-126, Carnegie-Mellon University, Pittsburgh, PA, 1986.
16. A. J. Robinson and F. Fallside. The utility driven dynamic error propagation network. Technical Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department, 1987.
17. 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2):243-248, 1992.
18. H. T. Siegelmann and E. D. Sontag. Turing computability with neural nets. Applied Mathematics Letters, 4(6):77-80, 1991.
19. S. Townley, A. Ilchmann, M. G. Weiss, W. McClements, A. C. Ruiz, D. Owens, and D. Praetzel-Wolters. Existence and learning of oscillations in recurrent neural networks. Technical Report AGTM 202, Universitaet Kaiserslautern, Fachbereich Mathematik, Kaiserslautern, Germany, 1999.
20. Fu-Sheng Tsung and Garrison W. Cottrell. Phase-space learning. In Advances in Neural Information Processing Systems, volume 7, pages 481-488. The MIT Press, 1995.
21. M. G. Weiss. Learning oscillations using adaptive control. Technical Report AGTM 178, Universitaet Kaiserslautern, Fachbereich Mathematik, Kaiserslautern, Germany, 1999.
22. R. J. Williams and J. Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 2(4):490-501, 1990.
23. R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent net works. Neural Computation, 1(2):270-280, 1989.
24. R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin and D. E. Rumelhart, editors, Back-propagation: Theory, Architectures and Applications, chapter 13, pages 433-486. Hillsdale, NJ: Erlbaum, 1992.