Learning long-term dependencies in NARX recurrent neural networks

IEEE Transactions on Neural Networks

Volumn 7, Issue 6, 1996, Pages 1329-1338

  Lin, Tsungnan ^b,c   Horne, Bill G ^a,b   Tiňo, Peter ^b,d   Giles, C Lee ^a,b,e  

^a IEEE

^b NEC LABORATORIES AMERICA   (United States)

^c Princeton University   (United States)

^d SLOVAK UNIVERSITY OF TECHNOLOGY   (Slovakia)

^e UNIVERSITY OF MARYLAND   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33646241633     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.548162     Document Type: Article

References (38)

1. A. D. Back and A. C. Tsoi, "FIR and IIR synapses, a new neural network architecture for time series modeling," Neural Computa., vol. 3, no. 3, pp. 375-385, 1991.
2. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient is difficult," IEEE Trans. Neural Networks, vol. 5, pp. 157-166, 1994.
3. S. Chen, S. A. Billings, and P. M. Grant, "Nonlinear system identification using neural networks," Int. J. Contr., vol. 51, no. 6, pp. 1191-1214, 1990.
4. J. Connor, L. E. Atlas, and D. R. Martin, "Recurrent networks and NARMA modeling." in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 301-308.
5. E. Coven, I. Kan, and J. Yorke, "Pseudo-orbit shadowing in the family of tent maps," Trans. Amer. Med. Soc., vol. 308, pp. 227-241, 1988.
6. B. de Vries and J. C. Principe, "The gamma model - A new neural model for temporal processing," Neural Networks, vol. 5, pp. 565-576, 1992.
7. J. E. Elman, "Finding structure in time," Cognitive Sci., vol. 14, pp. 179-211, 1990.
8. P. Frasconi, M. Gori, M. Maggini, and G. Soda, "Unified integration of explicit rules and learning by example in recurrent networks," IEEE Trans. Knowledge Data Eng., vol. 7, pp. 340-346, 1995.
9. P. Frasconi, M. Gori, and G. Soda, "Local feedback multilayered networks," Neural Computa. vol. 4, pp. 120-130, 1992.
10. M. Garzon and F. Botelho, "Observability of neural-network behavior," in Advances in Neural Information Processing Systems 6, J. D. Cowen, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 455-462.
11. C. L. Giles and B. G. Horne, "Representation and learning in recurrent neural-network architectures," in Proc. 8th Yale Wkshp. Adaptive Learning Syst., 1994, pp. 128-134.
12. M. Gori, M. Maggini, and G. Soda, "Scheduling of modular architectures for inductive inference of regular grammars," in Proc. ECAI'94 Wkshp. Combining Symbolic Connectionist Procedures, Amsterdam, The Netherlands, Aug. 1994, pp. 78-87.
13. S. El Hihi and Y. Bengio, "Hierarchical recurrent neural networks for long-term dependencies," in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1996.
14. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Forschungsberichte Künstliche Intelligenz FKI-207-95, Institut für Informatik, Technische Universität München, Germany, 1995.
15. B. G. Horne and C. L. Giles, "An experimental comparison of recurrent neural networks," in Advances in Neural Information Processing Systems 7. Cambridge, MA: MIT Press, 1995, pp. 697-704.
16. T. Kailath, Linear Systems. Englewood Cliffs, NJ: Prentice-Hall, 1980.
17. H. K. Khalil, Nonlinear Systems. New York: Macmillan, 1992.
18. R. R. Leighton and B. C. Conrath, "The autoregressive backpropagation algorithm," in Proc. Int. Joint Conf. Neural Networks, vol. 2, July 1991, pp. 369-377.
19. I. J. Leontaritis and S. A. Billings, "Input-output parametric models for nonlinear systems: Part I: Deterministic nonlinear systems," Int. J. Contr., vol. 41, no. 2, pp. 303-328, 1985.
20. L. Ljung, System Identification: Theory for the User. Englewood Cliffs, NJ: Prentice-Hall, 1987.
21. M. C. Mozer, "Induction of multiscale temporal structure," in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 275-282.
22. K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 4-27, Mar. 1990.
23. O. Nerrand, P. Roussel-Ragot, L. Personnaz, G. Dreyfus, and S. Marcos, "Neural networks and nonlinear adaptive filtering: Unifying concepts and new algorithms," Neural Computa., vol. 5, pp. 165-199, 1993.
24. P. Poddar and K. P. Unnikrishnan, "Nonlinear prediction of speech signals using memory neuron networks," in Proc. 1991 IEEE Wkshp. Neural Networks for Signal Processing, 1991, pp. 1-10.
25. G. V. Puskorius and L. A. Feldkamp, "Neurocontrol of nonlinear dynamical systems with Kalman filter-trained recurrent networks," IEEE Trans. Neural Networks, vol. 5, pp. 279-297, 1994.
26. S.-Z. Qin, H.-T. Su, and T. J. McAvoy, "Comparison of four neural-net learning methods for dynamic system identification," IEEE Trans. Neural Networks, vol. 3, pp. 122-130, 1992.
27. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, pp. 318-362.
28. J. Schmidhuber, "Learning complex, extended sequences using the principle of history compression," Neural Computa., vol. 4, no. 2, pp. 234-242, 1992.
29. _, "Learning unambiguous reduced sequence descriptions," in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 291-298.
30. D. R. Seidl and D. Lorenz, "A structure by which a recurrent neural network can approximate a nonlinear dynamic system," in Proc. Int. Joint Conf. Neural Networks, vol. 2, July 1991, pp. 709-714.
31. H. T. Siegelmann, B. G. Horne, and C. L. Giles, "Computational capabilities of recurrent narx neural networks," IEEE Trans. Syst., Man, Cybern., to be published. Also, Univ. Maryland, College Park, MD, Tech. Rep. UMIACS-TR-95-78 and CS-TR-3500.
32. H. T. Siegelmann and E. D. Sontag, "On the computational power of neural networks," J. Comput. Syst. Sci., vol. 50, no. 1, pp. 132-150, 1995.
33. E. D. Sontag, "Systems combining linearity and saturations and relations to neural networks," Rutgers Center for Syst. Contr., Tech. Rep. SYCON-92-01, 1992.
34. H.-T. Su and T. J. McAvoy, "Identification of chemical processes using recurrent networks," in Proc. Amer. Contr. Conf., vol. 3, 1991, pp. 2314-2319.
35. H.-T. Su, T. J. McAvoy, and P. Werbo, "Long-term predictions of chemical processes using recurrent neural networks: A parallel training approach," Ind. Eng. Chem. Res., vol. 31, pp. 1338-1352, 1992.
36. S. T. Venkataraman, "On encoding nonlinear oscillations in neural networks for locomotion," in Proc. 8th Yale Wkshp. Adaptive Learning Syst.s, 1994, pp. 14-20.
37. E. A. Wan, "Time series prediction by using a connectionist network with internal delay lines," in Time Series Prediction, A. S. Weigend and N. A. Gershenfeld, Eds. Reading, MA: Addison-Wesley, 1994, pp. 195-217.
38. R. J. Williams and D. Zipser, "Gradient-based learning algorithms for recurrent networks and their computational complexity," in Backpropagation: Theory, Architectures, and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Hillsdale, NJ: Lawrence Erlbaum, 1995, ch. 13, pp. 433-486.