Learning long-term dependencies with recurrent neural networks

Neurocomputing

Volumn 71, Issue 13-15, 2008, Pages 2481-2488

  Schaefer, Anton Maximilian ^a,b   Udluft, Steffen ^a   Zimmermann, Hans Georg ^a  

^a SIEMENS AG   (Germany)

^b UNIVERSITY OF OSNABRÜCK   (Germany)

Author keywords

Backpropagation; Inflation; Long term dependencies; Memory; Recurrent neural networks; State space model; System identification; Vanishing gradient

Indexed keywords

BACKPROPAGATION; DATA STORAGE EQUIPMENT; DYNAMICAL SYSTEMS; IDENTIFICATION (CONTROL SYSTEMS); LONG SHORT-TERM MEMORY; STATE SPACE METHODS; TIME DELAY;

COUNTER EXAMPLES; INFLATION; LONG AND SHORT TERM MEMORY; LONG-TERM DEPENDENCIES; OPEN DYNAMICAL SYSTEMS; RECURRENT NEURAL NETWORK (RNN); STATE - SPACE MODELS; VANISHING GRADIENT;

RECURRENT NEURAL NETWORKS;

ANALYTICAL ERROR; ARTIFICIAL NEURAL NETWORK; CONFERENCE PAPER; CONTROLLED STUDY; EMPIRICISM; LEARNING ALGORITHM; LONG TERM MEMORY; MATHEMATICAL COMPUTING; NONLINEAR SYSTEM; PRIORITY JOURNAL; RECURRENT NEURAL NETWORK; SENSITIVITY ANALYSIS; SHORT TERM MEMORY; STOCHASTIC MODEL; TIME;

EID: 56449109755     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neucom.2007.12.036     Document Type: Conference Paper

References (19)

1. Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Trans. Neural Networks 5 (2) (1994) 157-166.
2. K.-L. Du, M.N.S. Swamy, Neural Networks in a Softcomputing Framework, Springer, London, 2006.
3. S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan, New York, 1994.
4. S. Haykin, J. Principe, T. Sejnowski, J. McWhirter, New Directions in Statistical Signal Processing: From Systems to Brain, MIT Press, Cambridge, MA, 2006.
5. S. Hochreiter, The vanishing gradient problem during learning recurrent neural nets and problem solutions, Int. J. Uncertainty Fuzziness Knowl. Syst. 6 (2) (1998) 107-116.
6. S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, Gradient flow in recurrent nets: The difficulty of learning long-term dependencies, in: J.F. Kolen, S. Kremer (Eds.), A Field Guide to Dynamical Recurrent Networks, IEEE Press, New York, 2001, pp. 237-243.
7. S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (8) (1997) 1735-1780.
8. K. Hornik, M. Stinchcombe, H. White, Multi-layer feedforward networks are universal approximators, Neural Networks 2 (1989) 359-366.
9. J.F. Kolen, S.C. Kremer, A Field Guide to Dynamical Recurrent Networks, IEEE Press, New York, 2001.
10. L.R. Medsker, L.C. Jain, Recurrent Neural Networks: Design and Application, Computer Intelligence, vol. 1, CRC Press international, 1999.
11. R. Neuneier, H.G. Zimmermann, How to train neural networks, in: G.B. Orr, K.-R. Mueller (Eds.), Neural Networks: Tricks of the Trade, Springer, Berlin, 1998, pp. 373-423.
12. B. Pearlmutter, Gradient calculations for dynamic recurrent neural networks: A survey, IEEE Trans. Neural Networks 6 (5) (1995) 1212-1228.
13. D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning internal representations by error propagation, in: D.E. Rumelhart, et al. (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, MIT Press, Cambridge, MA, 1986, pp. 318-362.
14. A.M. Schaefer, H.G. Zimmermann, Recurrent neural networks are universal approximators, Int. J. Neural Systems 17 (4) (2007) 253-263.
15. A. Soofi, L. Cao, Modeling and Forecasting Financial Data, Techniques of Nonlinear Dynamics, Kluwer Academic Publishers, Dordrecht, 2002.
16. P.J. Werbos, The Roots of Backpropagation. From Ordered Derivatives to Neural Networks and Political Forecasting, Wiley, New York, 1994.
17. R.J. Williams, D. Zipser, Gradient-based learning algorithms for recurrent connectionist networks, in: Y. Chauvin, D.E. Rumelhart (Eds.), Backpropagation: Theory, Architectures, and Applications, Erlbaum, Hillsdale, NJ, 1990.
18. H.G. Zimmermann, R. Grothmann, A.M. Schaefer, C. Tietz, Identification and forecasting of large dynamical systems by dynamical consistent neural networks, in: S. Haykin, J. Principe, T. Sejnowski, J. McWhirter (Eds.), New Directions in Statistical Signal Processing: From Systems to Brain, MIT Press, Cambridge, MA, 2006, pp. 203-242.
19. H.G. Zimmermann, R. Neuneier, Neural network architectures for the modeling of dynamical systems, in: J.F. Kolen, S. Kremer (Eds.), A Field Guide to Dynamical Recurrent Networks, IEEE Press, New York, 2001, pp. 311-350.