How embedded memory in recurrent neural network architectures helps learning long-term temporal dependencies

Neural Networks

Volumn 11, Issue 5, 1998, Pages 861-868

  Lin, Tsungnan ^a   Horne, Bill G ^b   Giles, C Lee ^c,d  

^a Epson Palo Alto Laboratory   (United States)

^b AADM Consulting   (United States)

^c NEC LABORATORIES AMERICA   (United States)

^d UNIVERSITY OF MARYLAND   (United States)

Author keywords

Automata; Gradient descent; Latching; Long term dependencies; Memory; Neural networks; Recurrent; Time delay; Training

Indexed keywords

AUTOMATA THEORY; COMPUTATION THEORY; COMPUTER ARCHITECTURE; LEARNING SYSTEMS; STORAGE ALLOCATION (COMPUTER);

EMBEDDED MEMORY; GRADIENT DESCENT; LATCHING; LONG TERM DEPENDENCIES; TIME DELAY;

RECURRENT NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER MEMORY; DECISION THEORY; NONLINEAR SYSTEM; PRIORITY JOURNAL;

EID: 0032123428     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(98)00018-5     Document Type: Article

References (31)

1. Back A., Tsoi A. FIR and IIR synapses, a new neural network architecture for time series modeling. Neural Computation. 3(3):1991;337-350.
2. Bengio Y., Simard P., Frasconi P. Learning long-term dependencies with gradient is difficult. IEEE Transactions on Neural Networks. 5(2):1994;157-166.
3. Chen S., Billings S., Grant P. Non-linear system identification using neural networks. International Journal of Control. 51(6):1990;1191-1214.
4. de Vries B., Principe J.C. The gamma model - a new neural model for temporal processing. Neural Networks. 5:1992;565-576.
5. Elman J. Finding structure in time. Cognitive Science. 14:1990;179-211.
6. Frasconi P., Gori M. Computational capabilities of local-feedback recurrent networks acting as finite-state machines. IEEE Transactions on Neural Networks. 7(6):1996;1521-1524.
7. Frasconi P., Gori M., Soda G. Local feedback multilayered networks. Neural Computation. 4:1992;120-130.
8. Frasconi P., Gori M., Maggini M., Soda G. Unified integration of explicit rules and learning by example in recurrent networks. IEEE Transactions on Knowledge and Data Engineering. 7(2):1995;340-346.
9. Giles, C., Omlin, C., 1992. Inserting rules into recurrent neural networks. In: Kung, S., Fallside, F., Sorenson, J. A., Kamm, C. (Eds), Neural Network for Signal Processing II, Proceedings of the 1992 IEEE Workshop, Piscataway, NJ. IEEE Press, pp. 13-22.
10. Giles, C., Kuhn, G., Williams, R., 1994. Dynamic Recurrent Neural Networks: Theory and Applications. IEEE Transactions on Neural Networks, 5(2), Special Issue.
11. Gori, M., Maggini, M., Soda, G., 1994. Scheduling of modular architectures for inductive inference of regular grammars. In: ECAI'94 Workshop on Combining Symbolic and Connectionist Processing, Amsterdam. Wiley, Chichester, pp. 78-87.
12. Hihi, S. E., Bengio, Y., 1996. Hierarchical recurrent neural networks for long-term dependencies. In: Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA.
13. Hochreiter, S., Schmidhuber, J., 1997. Long short-term memory. Neural Computation, 9, p. 1681.
14. Horne, B., Giles, C., 1995. An experimental comparison of recurrent neural networks. In: Advances in Neural Information Processing Systems 7. MIT Press, Cambridge, MA, pp. 697-704.
15. Jordan, M., 1986. Attractor dynamics and parallelism in a connectionist sequential machine. In: Proceedings of the Ninth Annual conference of the Cognitive Science Society. Lawrence Erlbaum, pp. 531-546.
16. Lin T., Horne B., Tino P., Giles C. Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks. 7(6):1996;1329-1338.
17. Lin, T., Horne, B., Tino, P., Giles, C., 1996b. Learning long-term dependencies is not as difficult with narx recurrent neural networks. In: Advances in Neural Information Processing Systems 8. MIT Press, Cambridge, MA.
18. Lin T., Giles C., Horne B., Kung S. A delay damage model selection algorithm for narx neural networks. IEEE Transactions on Signal Processing. 45(11):1997;2719-2730. (Special Issue on Neural Networks).
19. Ljung, L., 1987. System Identification: Theory for the User. Prentice-Hall, Englewood Cliffs, NJ.
20. Mozer, M.C., 1992. Induction of multiscale temporal structure. In: Moody, J., Hanson, S.J., Lippmann, R. (Eds.), Neural Information Processing Systems 4. Morgan Kaufmann, pp. 275-282.
21. Mozer, M.C., 1994. Neural net architectures for temporal sequence processing. In: Weigend, A., Gershenfeld, N. (Eds.), Time Series Prediction. Addison-Wesley, Reading, MA, pp. 243-264.
22. Narendra K., Parthasarathy K. Identification and control of dynamical systems using neural networks. IEEE Transactions on Neural Networks. 1(1):1990;4-27.
23. Omlin C., Giles C. Rule revision with recurrent neural networks. IEEE Transactions on Knowledge and Data Engineering. 8(1):1996;183-188.
24. Pedersen, M., Hansen, L., 1995. Recurrent networks: second order properties and pruning. In: Tesauro, G., Touretzky, D., Leen, T. (Eds.), Advances in Neural Information Processing Systems 7. MIT Press, Cambridge, MA.
25. Sastry P., Santharam G., Unnikrishnan K. Memory neuron networks for identification and control of dynamical systems. IEEE Transactions on Neural Networks. 5(2):1994;306-319.
26. Seidl, D., Lorenz, R., 1991. A structure by which a recurrent neural network can approximate a nonlinear dynamic system. In: Proceedings of the International Joint Conference on Neural Networks 1991, Vol. II, pp. 709-714.
27. Siegelmann H., Sontag E. On the computational power of neural nets. Journal of Computer and System Sciences. 50(1):1995;132-150.
28. Siegelmann H., Horne B., Giles C. Computational capabilities of recurrent narx neural networks. IEEE Transactions on Systems, Man and Cybernetics - Part B. 27(2):1997;208.
29. Su, H.-T., McAvoy, T., 1991. Identification of chemical processes using recurrent networks. In: Proceedings of the American Controls Conference, Vol. 3, pp. 2314-2319.
30. Su H.-T., McAvoy T., Werbos P. Long-term predictions of chemical processes using recurrent neural networks: a parallel training approach. Industrial Engineering and Chemical Research. 31:1992;1338-1352.
31. Tsoi A., Back A. Locally recurrent globally feedforward networks, a critical review of architectures. IEEE Transactions on Neural Networks. 5(2):1994;229-239.