Learning Long-Term Dependencies with Gradient Descent is Difficult

IEEE Transactions on Neural Networks

Volumn 5, Issue 2, 1994, Pages 157-166

  Bengio, Yoshua ^a,b   Simard, Patrice ^b   Frasconi, Paolo ^c  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b LUCENT TECHNOLOGIES   (United States)

^c UNIVERSITY OF FLORENCE   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; BINARY SEQUENCES; COMPUTATIONAL COMPLEXITY; DATA STORAGE EQUIPMENT; DISCRETE TIME CONTROL SYSTEMS; INFORMATION RETRIEVAL SYSTEMS; INPUT OUTPUT PROGRAMS; LEARNING SYSTEMS; PARAMETER ESTIMATION;

GRADIENT BASED LEARNING ALGORITHMS; INFORMATION LATCHING; INPUT/OUTPUT SEQUENCES; LEARNING ALGORITHMS; PARAMETRIC DYNAMICAL SYSTEM; RECURRENT NEURAL NETWORK TRAINING; TEMPORAL CONTINGENCIES;

NEURAL NETWORKS;

EID: 0028392483     PISSN: 10459227     EISSN: 19410093     Source Type: Journal    
DOI: 10.1109/72.279181     Document Type: Article

References (23)

1. P. L. Bartlett, and T. Downs, “Using Random Weights to train Multilayer Networks of Hard-Limiting Units.” IEEE Transactions on Neural Networks, vol. 3, no. 2, 1992, pp. 202–210.
2. S. Becker and Y. Le Cun, “Improving the convergence of backpropagation learning with second order methods,” Proceedings of the 1988 Connectionist Models Summer School, Touretzky, Hinton, and Sejnowski, Eds. San Matteo, CA: Morgan Kaufmann, pp. 29–37.
3. Y. Bengio, P. Frasconi, P. Simard, “The problem of learning long-term dependencies in recurrent networks,” invited paper at the IEEE International Conference on Neural Networks 1993, San Francisco, IEEE Press, pp. 1183–1188
4. Y. Bengio, “Artificial neural networks and their application to sequence recognition,” Ph.D. thesis, McGill University, 1991, Montreal, Quebec, Canada.
5. Y. Bengio, R. De Mori, G. Flammia, and R. Kompe, “Global optimization of a neural network—Hidden Markov model hybrid,” IEEE Transactions on Neural Networks, vol. 3, no. 2, 1992, pp. 252–259.
6. A. Corana, M. Marchesi, C. Martini, and S. Ridella, “Minimizing multimodal functions of continuous variables with the simulated annealing algorithm,” ACM Transactions on Mathematical Software, vol. 13, no. 13, Sept. 1987, pp. 262–280.
7. P. Frasconi, M. Gori, and G. Soda, “Local Feedback Multilayered Networks,” Neural Computation 4(1), 1992, pp. 120–130.
8. P. Frasconi, M. Gori, M. Maggini, and G. Soda, “Unified integration of explicit rules and learning by example in recurrent networks,” IEEE Trans. on Knowledge and Data Engineering, in press.
9. R. J. Gaynier, and T. Downs, “A method of training multi-layer networks with heaviside characteristics using internal representations,” IEEE International Conference on Neural Networks 1993, San Francisco, pp. 1812–1817.
10. M. Gori, Y. Bengio, and R. De Mori, “BPS: a learning algorithm for capturing the dynamic nature of speech,” Proc. IEEE Int. Joint Conf. on Neural Networks, Washington, DC, 1989, pp. II.417-II.424.
11. C. L. Giles and C. W. Omlin, “Inserting rules into recurrent neural networks,” in Neural Networks for Signal Processing II, Proceedings of the 1992 IEEE workshop, Kung, Fallside, Sorenson, and Kamm, Eds. IEEE Press, pp. 13–22.
12. T. Grossman, R. Meir, and E. Domany, “Learning by choice of internal representation,” Neural Information Processing Systems I. D. S. Touretzky, Ed. pp. 73–80.
13. S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, “Optimization by simulated annealing,” Science 220, pp.671–680, 4598 (May 1983).
14. G. Kuhn, “A first look at phonetic discrimination using connectionist models with recurrent links.” CCRP — IDA SCIMP working paper No.4/87, Institute for Defense Analysis, Princeton, NJ, 1987.
15. K. J. Lang and G. E. Hinton, “The development of the Time-Delay Neural Network architecture for speech recognition,” Technical Report CMU-CS-88-152, Carnegie-Mellon University, 1988.
16. Y. Le Cun, “Learning processes in an asymmetric threshold network,” in Disordered systems and biological organization,E. Bienenstock, F. Fogelman-Soulié, and G. Weisbuch, Eds. Les Houches, France: Springer-Verlag, 1986, pp. 233–240.
17. C. M. Marcus, F. R. Waugh, and R. M. Westervelt, “Nonlinear dynamics and stability of analog neural networks,” Physica D 51 (special issue), pp. 234—247, 1991.
18. M. C. Mozer, “A focused back-propagation algorithm for temporal pattern recognition,” Complex Systems, 3, pp. 349–391, 1989.
19. M. C. Mozer, “Induction of multiscale temporal structure,” in Advances in Neural Information Processing Systems 4, Moody, Hanson, and Lippman, Eds. San Matteo, CA: Morgan Kaufmann, 1992, pp. 275–282.
20. J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Non-linear Equations in Several Variables and Systems of Equations, New York: Academic Press, 1960.
21. R. Rohwer, “The ‘moving targets’ training algorithm,” Advances in Neural Information Processing Systems 2, Touretzky, Ed. San Matteo, CA: Morgan Kaufmann, 1990, pp. 558–565.
22. D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning internal representation by error propagation,” in Parallel Distributed Processing, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, Bradford Books, vol. 1, 1986, pp. 318–362.
23. R. J. Williams and D. Zipser “A learning algorithm for continuously running fully recurrent neural networks,” Neural Computation, 1, 1989, pp. 270–280.