Long Short-Term Memory

Neural Computation

Volumn 9, Issue 8, 1997, Pages 1735-1780

  Hochreiter, Sepp ^a   Schmidhuber, Jürgen ^b  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; LEARNING; MEMORY; NERVE CELL NETWORK; PHYSIOLOGY; PSYCHOLOGICAL MODEL; SHORT TERM MEMORY; TIME;

ALGORITHMS; LEARNING; MEMORY; MEMORY, SHORT-TERM; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; NERVE NET; NEURAL NETWORKS (COMPUTER); TIME FACTORS;

EID: 0031573117     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/neco.1997.9.8.1735     Document Type: Article

References (42)

1. Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In IEEE 1st International Conference on Neural Networks, San Diego (Vol. 2, pp. 609-618).
2. Baldi, P., & Pineda, F. (1991). Contrastive learning and neural oscillator. Neural Computation, 3, 526-545.
3. Bengio, Y., & Frasconi, P. (1994). Credit assignment through time: Alternatives to backpropagation. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems 6 (pp. 75-82). San Mateo, CA: Morgan Kaufmann.
4. Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166.
5. Cleeremans, A., Servan-Schreiber, D., & McClelland, J. L. (1989). Finite-state automata and simple recurrent networks. Neural Computation, 1, 372-381.
6. de Vries, B., & Principe, J. C. (1991). A theory for neural networks with time delays. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3, (pp. 162-168). San Mateo, CA: Morgan Kaufmann.
7. Doya, K. (1992). Bifurcations in the learning of recurrent neural networks. In Proceedings of 1992 IEEE International Symposium on Circuits and Systems (pp. 2777-2780).
8. Doya, K., & Yoshizawa, S. (1989). Adaptive neural oscillator using continuous-time backpropagation learning. Neural Networks, 2, 375-385.
9. Elman, J. L. (1988). Finding structure in time (Tech. Rep. No. CRL 8801). San Diego: Center for Research in Language, University of California, San Diego.
10. Fahlman, S. E. (1991). The recurrent cascade-correlation learning algorithm. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 190-196). San Mateo, CA: Morgan Kaufmann.
11. Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München. See http://www7.informatik.tu-muenchen.de/∼hochreit.
12. Hochreiter, S., & Schmidhuber, J. (1995). Long short-term memory (Tech. Rep. No. FKI-207-95). Fakultät für Informatik, Technische Universität München.
13. Hochreiter, S., & Schmidhuber, J. (1996). Bridging long time lags by weight guessing and "long short-term memory." In F. L. Silva, J. C. Principe, & L. B. Almeida (Eds.), Spatiotemporal models in biological and artificial systems (pp. 65-72). Amsterdam: IOS Press.
14. Hochreiter, S., & Schmidhuber, J. (1997). LSTM can solve hard long time lag problems. In Advances in neural information processing systems 9. Cambridge, MA: MIT Press.
15. Lang, K., Waibel, A., & Hinton, G. E. (1990). A time-delay neural network architecture for isolated word recognition. Neural Networks, 3, 23-43.
16. Lin, T., Horne, B. G., Tino, P., & Giles, C. L. (1996). Learning long-term dependencies in NARX recurrent neural networks. IEEE Transactions on Neural Networks, 7, 1329-1338.
17. Miller, C. B., & Giles, C. L. (1993). Experimental comparison of the effect of order in recurrent neural networks. International Journal of Pattern Recognition and Artificial Intelligence, 7(4), 849-872.
18. Mozer, M. C. (1989). A focused back-propagation algorithm for temporal sequence recognition. Complex Systems, 3, 349-381.
19. Mozer, M. C. (1992). Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 275-282). San Mateo, CA: Morgan Kaufmann.
20. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2), 263-269.
21. Pearlmutter, B. A. (1995). Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural Networks, 6(5), 1212-1228.
22. Pineda, F. J. (1987). Generalization of back-propagation to recurrent neural networks. Physical Review Letters, 19(59), 2229-2232.
23. Pineda, F. J. (1988). Dynamics and architecture for neural computation. Journal of Complexity, 4, 216-245.
24. Plate, T. A. (1993). Holographic recurrent networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles ( Eds.), Advances in neural information processing systems 5 (pp. 34-41). San Mateo, CA: Morgan Kaufmann.
25. Pollack, J. B. (1991). Language induction by phase transition in dynamical recognizers. In R. P. Lippmann, J. E. Moody, & D. S. Touretzky (Eds.), Advances in neural information processing systems 3 (pp. 619-626). San Mateo, CA: Morgan Kaufmann.
26. Puskorius, G. V., and Feldkamp, L. A. (1994). Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2), 279-297.
27. Ring, M. B. (1993). Learning sequential tasks by incrementally adding higher orders. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 115-122). San Mateo, CA: Morgan Kaufmann.
28. Robinson, A. J., & Fallside, F. (1987). The utility driven dynamic error propagation network (Tech. Rep. No. CUED/F-INFENG/TR.1). Cambridge: Cambridge University Engineering Department.
29. Schmidhuber, J. (1989). A local learning algorithm for dynamic feedforward and recurrent networks. Connection Science, 1(4), 403-412.
30. 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation, 4(2), 243-248.
31. Schmidhuber, J. (1992b). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2), 234-242.
32. Schmidhuber, J. (1992c). Learning unambiguous reduced sequence descriptions. In J. E. Moody, S. J. Hanson, & R. P. Lippman (Eds.), Advances in neural information processing systems 4 (pp. 291-298). San Mateo, CA: Morgan Kaufmann.
33. Schmidhuber, J. (1993). Netzwerkarchitekturen, Zielfunktionen und Kettenregel. Habilitationsschrift, Institut für Informatik, Technische Universität München.
34. Schmidhuber, J., & Hochreiter, S. (1996). Guessing can outperform many long time lag algorithms (Tech. Rep. No. IDSIA-19-96). Lugano, Switzerland: Instituto Dalle Molle di Studi sull'Intelligenza Artificiale.
35. Silva, G. X., Amaral, J. D., Langlois, T., & Almeida, L. B. (1996). Faster training of recurrent networks. In F. L. Silva, J. C. Principe, & L. B. Almeida (Eds.), Spatiotemporal models in biological and artificial systems (pp. 168-175). Amsterdam: IOS Press.
36. Smith, A. W., & Zipser, D. (1989). Learning sequential structures with the real-time recurrent learning algorithm. International Journal of Neural Systems, 1(2), 125-131.
37. Sun, G., Chen, H., & Lee, Y. (1993). Time warping invariant neural networks. In S. J. Hanson, J. D. Cowan, & C. L. Giles (Eds.), Advances in neural information processing systems 5 (pp. 180-187). San Mateo, CA: Morgan Kaufmann.
38. Watrous, R. L., & Kuhn, G. M. (1992). Induction of finite-state languages using second-order recurrent networks. Neural Computation, 4, 406-414.
39. Werbos, P. J. (1988). Generalization of backpropagation with application to a recurrent gas market model. Neural Networks, 1, 339-356.
40. Williams, R. J. (1989). Complexity of exact gradient computation algorithms for recurrent neural networks (Tech. Rep. No. NU-CCS-89-27). Boston: Northeastern University, College of Computer Science.
41. Williams, R. J. & Peng, J. (1990). An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 4, 491-501.
42. Williams, R. J., & Zipser, D. (1992). Gradient-based learning algorithms for recurrent networks and their computational complexity. In Y. Chauvin, & D. E. Rumelhart (Eds.), Back-propagation: Theory, architectures and applications. Hillsdale, NJ: Erlbaum.