The vanishing gradient problem during learning recurrent neural nets and problem solutions

International Journal of Uncertainty, Fuzziness and Knowldege-Based Systems

Volumn 6, Issue 2, 1998, Pages 107-116

  Hochreiter, Sepp ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Long Short Term Memory; Long term dependencies; Recurrent neural nets; Vanishing gradient

Indexed keywords

EID: 0042276525     PISSN: 02184885     EISSN: None     Source Type: Journal    
DOI: 10.1142/S0218488598000094     Document Type: Article

References (30)

1. S. Hochreiter and J. Schmidhuber, "Long short-term memory", Neural Computation, 9(8):1735-1780 (1997).
2. G. V. Puskorius and L. A. Feldkamp, "Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks", IEEE Transactions on Neural Networks, 5(2):279-297 (1994).
3. M. C. Mozer, "Induction of multiscale temporal structure", in Advances in Neural Information Processing Systems 4, ed. J. E. Moody et al., (Morgan Kaufmann, San Mateo, 1992), pages 275-282.
4. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity", in Back-propagation: Theory, Architectures and Applications, ed. Y. Chauvin and D. E. Rumelhart (Hillsdale, Erlbaum, New York, 1992).
5. A. J. Robinson and F. Fallside, "The utility driven dynamic error propagation network", Technical Report CUED/F-INFENG/TR.1, Cambridge Univ. Engineering Department, 1987.
6. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult", IEEE Transactions on Neural Networks, 5(2):157-166 (1994).
7. S. Hochreiter, "Untersuchungen zu dynamischen neuronalen Netzen", Diploma Thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Univ. München, 1991. See www7.informatik.tu-muenchen.de/∼hochreit.
8. G. Deco and W. Brauer, "Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures", Neural Networks, 8(4):525-535 (1995).
9. F. J. Pineda, "Dynamics and architecture for neural computation", Journal of Complexity, 4:216-245 (1988).
10. S. Hochreiter and J. Schmidhuber, "LSTM can solve hard long time lag problems", in Advances in Neural Information Processing Systems 9, ed. M. C. Mozer et al. (MIT Press, Cambridge MA, 1997), pages 473-479.
11. S. Hochreiter and J. Schmidhuber, "Bridging long time lags by weight guessing and Long Short-Term Memory", in Spatiotemporal models in biological and artificial systems, ed. F. L. Silva et al. (IOS Press, Amsterdam, Netherlands, 1996), pages 65-72.
12. J. L. Elman, "Finding structure in time", Technical Report CRL 8801, Center for Research in Language, Univ. of California, San Diego, 1988.
13. S. E. Fahlman, "The recurrent cascade-correlation learning algorithm", in Advances in Neural Information Processing Systems 3, ed. R. P. Lippmann et al. (Morgan Kaufmann, San Mateo, 1991), pages 190-196.
14. R. J. Williams, "Complexity of exact gradient computation algorithms for recurrent neural networks", Technical Report NU-CCS-89-27, Boston: Northeastern Univ., College of Computer Science, 1989.
15. 3) time complexity learning algorithm for fully recurrent continually running networks", Neural Computation, 4(2):243-248 (1992).
16. B. A. Pearlmutter, "Learning state space trajectories in recurrent neural networks", Neural Computation, 1(2):263-269 (1989).
17. B. A. Pearlmutter, "Gradient calculations for dynamic recurrent neural networks: A survey", IEEE Transactions on Neural Networks, 6(5):1212-1228 (1995).
18. J. Schmidhuber and S. Hochreiter, "Guessing can outperform many long time lag algorithms", Technical Report IDSIA-19-96, IDSIA, 1996.
19. S. Hochreiter and J. Schmidhuber, "Flat minima", Neural Computation, 9(1):1-42 (1997).
20. Y. Bengio and P. Frasconi, "Credit assignment through time: Alternatives to backpropagation", in Advances in Neural Information Processing Systems 6, ed. J. D. Cowan et al. (Morgan Kaufmann, San Mateo, 1994), pages 75-82.
21. J. Schmidhuber, "Learning complex, extended sequences using the principle of history compression", Neural Computation, 4(2):234-242 (1992).
22. R. L. Watrous and G. M. Kuhn, "Induction of finite-state languages using second-order recurrent networks", Neural Computation, 4:406-414 (1992).
23. C. B. Miller and C. L. Giles, "Experimental comparison of the effect of order in recurrent neural networks", International journal of Pattern Recognition and Artificial Intelligence, 7(4):849-872 (1993).
24. K. Lang, A. Waibel, and G. E. Hinton, "A time-delay neural network architecture for isolated word recognition", Neural Networks, 3:23-43 (1990).
25. T. Lin, B. G. Horne, P. Tino, and C. L. Giles, "Learning long-term dependencies in NARX recurrent neural networks", IEEE Transactions on Neural Networks, 7(6):1329-1338 (1996).
26. T. A. Plate, "Holographic recurrent networks", in Advances in Neural Information Processing Systems 5, ed. J. D. Cowan et al. (Morgan Kaufmann, San Mateo, 1993), pages 34-41.
27. B. de Vries and J. C. Principe, "A theory for neural networks with time delays", in Advances in Neural Information Processing Systems 3, ed. R. P. Lippmann et al. (Morgan Kaufmann, San Mateo, 1991), pages 162-168.
28. G. Sun, H. Chen, and Y. Lee, "Time warping invariant neural networks", in Advances in Neural Information Processing Systems 5, ed. J. D. Cowan et al. (Morgan Kaufmann, San Mateo, 1993), pages 180-187.
29. A. W. Smith and D. Zipser, "Learning sequential structures with the real-time recurrent learning algorithm", International Journal of Neural Systems, 1(2):125-131 (1989).
30. A. Cleeremans, D. Servan-Schreiber, and J. L. McClelland, "Finite-state automata and simple recurrent networks", Neural Computation, 1:372-381 (1989).