New results on recurrent network training: unifying the algorithms and accelerating convergence

IEEE Transactions on Neural Networks

Volumn 11, Issue 3, 2000, Pages 697-709

  Atiya, Amir F ^a,b   Parlos, Alexander G ^a,c  

^a IEEE   (United States)

^b California Institute of Technology   (United States)

^c Texas AandM University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BACKPROPAGATION THROUGH TIME; CONSTRAINED OPTIMIZATION; ERROR GRADIENT; REAL TIME RECURRENT LEARNING;

APPROXIMATION THEORY; BACKPROPAGATION; COMPUTATIONAL COMPLEXITY; DIFFERENTIAL EQUATIONS; ERROR ANALYSIS; LEARNING ALGORITHMS; MATRIX ALGEBRA; OPTIMAL CONTROL SYSTEMS; OPTIMIZATION; RECURSIVE FUNCTIONS;

RECURRENT NEURAL NETWORKS;

EID: 0034186923     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.846741     Document Type: Article

References (53)

1. A. Parlos, K. Chong, and A. Atiya, "Application of the recurrent multilayer perceptron in modeling complex process dynamics," IEEE Trans. Neural Networks, vol. 5, pp. 255-266, Mar. 1994.
2. J. Moody and L. Wu, "Optimization of trading systems and portfolios," in Proc. Neural Networks Capital Markets Conf. Pasadena, CA, Nov. 1996.
3. J. Connors, D. Martin, and L. Atlas, "Recurrent neural networks and robust time series prediction," IEEE Trans. Neural Networks, vol. 5, pp. 240-254, Mar. 1994.
4. D. Bassi, "Stock price predictions by recurrent multilayer neural network architectures," in Proc. Neural Networks in the Capital Markets Conf., A. Refenes, Ed.. London, U.K., October 1995, London Business School, pp. 331-340.
5. C. Omlin and L. Giles, "Exatraction of rules from discrete-time recurrent neural networks," Neural Networks, vol. 9, no. 1, pp. 41-52, 1996.
6. G. Puskorius and L. Feldman, "Neurocontrol of dynamical systems with Kalman filter trained recurrent networks," IEEE Trans. Neural Networks, vol. 5, pp. 279-297, Mar. 1994.
7. K. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks, "IEEE Trans. Neural Networks, vol. 1, pp. 4-27, 1990.
8. J. Suykens, B. De Moor, and J. Vandewalle, "Nonlinear system identification using neural state-space models, applicable to robust control design," Int. J. Contr., vol. 62, no. 1, pp. 129-152, 1995.
9. S. Haykin, Neural Networks: A Comprehensive Foundation: IEEE Press, 1994.
10. R. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Comput., vol. 1, pp. 270-280, 1989.
11. A. Robinson and F. Fallside, "The utility driven dynamic error propagation network," Cambridge Univ. Eng. Dept., Tech. Rep. CUED/F-IN-FENG/TR.1, 1987.
12. D. Rumelhart, G. Hinton, and R. Williams, "Learning internal representation by error propagation," Parallel Distibuted Processing, 1986.
13. P. Werbos, "Backpropagation through time: What it does and how to do it," Proc. IEEE, vol. 78, Oct. 1990.
14. B. Pearlmutter, "Learning state-space trajectories in recurrent neural networks," Neural Comput., vol. 1, pp. 263-269, 1989.
15. N. Toomarian and J. Barhen, "Adjoint-functions and temporal learning algorithms in neural networks," Advances in Neural Information Processing Systems, 3, pp. 113-120, 1991.
16. _, "Learning a trajectory using adjoint functions and teacher forcing," Neural Networks, vol. 5, no. 3, pp. 473-484, 1992.
17. G.-Z. Sun, H.-H. Chen, and Y.-C. Lee, "Green's function method for fast on-line learning algorithm of recurrent neural networks," Advances Neural Inform. Processing Syst. 4, pp. 333-340, 1992.
18. 3) time complexity learning algorithm for fully recurrent continually running networks," Neural Comput., vol. 4, no. 2, pp. 243-248, 1992.
19. R. Williams, "Complexity of exact gradient computation algorithms for recurrent neural networks," Northeastern Univ., College Comp. Sci., Boston, MA, Tech. Rep. NU-CCS-89-27, 1989.
20. R. Williams and J. Peng, "Gradient-based learning algorithms for recurrent networks and their computational complexity," in Backpropagation: Theory, Architectures, and Applications. Hillsdale, NJ: Lawrence Erlbaum, 1992.
21. _, "An efficient gradient-based algorithm for on-line training of recurrent network trajectories," Neural Comput., vol. 2, pp. 490-501, 1990.
22. J. Sum, L. Chan, C. Leung, and G. Young, "Extended Kalman filter-based pruning method for recurrent neural networks," Neural Comput., vol. 10, no. 6, pp. 1481-1506, 1998.
23. S. Ma and C. Ji, "Fast training of recurrent networks based on EM-algorithm," IEEE Trans. Neural Networks, vol. 9, pp. 11-26, Jan. 1998.
24. S. Ma and C. Ji, "A unified approach on fast training of feedforward and recurrent networks using EM algorithm," Trans. Signal Processing, vol. 46, pp. 2270-2274, Aug. 1998.
25. M. Mozer, "A focused back-propagation algorithm for temporal sequence recognition," Complex Syst., vol. 3, pp. 349-381, 1989.
26. J. Hochreiter, "Untersuchungen zu dynamischen neuronalen Netzen," Diploma Thesis, Institut für Informatik, Technische Universität München, 1991.
27. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, Nov. 1997.
28. F. Pineda, "Dynamics and architecture for neural computation," J. Complexity, vol. 4, pp. 216-245, 1988.
29. L. Almeida, "A learning rule for asynchronous perceptrons with feedback in a combinatotrial environment," in Proc. IEEE 1st Int. Conf. Neural Networks, vol. 2, San Diego, CA, 1987, pp. 609-618.
30. A. Atiya, "Learning on a general network," in Neural Inform. Processing Syst., D. Anderson, Ed. New York: Amer. Inst. Physics, 1988.
31. A. Atiya and A. Parlos, "Unifying recurrent network training algorithms," in Proc. World Congr. Neural Networks, Portland, OR, Jul. 1993.
32. P. Baldi, "Gradient descent learning algorithms: Ageneral overview," IEEE Trans. Neural Networks, vol. 6, pp. 182-195, Jan. 1995.
33. B. Pearlmutter, "Gradient calculation for dynamic recurrent neural networks - A survey," IEEE Trans. Neural Networks, vol. 6, pp. 1212-1228, Sept. 1995.
34. F. Pineda, "Time dependent adaptive neural networks," in Advances in Neural Information Processing Systems, 2, D. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 710-718.
35. F. Beaufays and E. Wan, "Relating real-time backpropagation and backpropagation-through-time: An application of flow graph interreciprocity," Neural Comput., vol. 6, pp. 296-306, 1994.
36. E. Wan and F. Beaufays, "Diagrammatic derivation of gradient algorithms forneural networks," Neural Comput., vol. 8, pp. 182-201, 1996.
37. A. Parlos, A. Atiya, K. Chong, W. Tsai, and B. Fernandez, "Recurrent multilayer perceptron for nonlinear system identification," in Proc. Int. Joint Neural Network Conf., Seattle, WA, Jul. 1991.
38. A. Tsoi and A. Back, "Locally recurrent globally feedforward networks - A critical review of architectures," IEEE Trans. Neural Networks, vol. 5, pp. 229-239, Mar. 1994.
39. A. Atiya and A. Parlos, "Identification of nonlinear dynamics using a general spatio-temporal network," Math. Comput. Modeling J., vol. 21, no. 1, pp. 53-71, Jan. 1995.
40. _, "Nonlinear system identification using spatio-temporal neural networks," in Proc. Int. Joint Neural Network Conf., Baltimore, MD, June 1992.
41. E. Wan, "Temporal backprogation for FIR neural networks," in Proc. Int. Joint Conf. Neural Networks, San Diego, CA, 1990, pp. 575-580.
42. P. Baldi and A. Atiya, "How delays affect neural dynamics and learning," IEEE Trans. Neural Networks, vol. 5, pp. 612-621, Jul. 1994.
43. q theory: Checking and imposing stability of recurrent neural networks for nonlinear modeling," IEEE Trans. Signal Processing, vol. 45, pp. 2682-2691, Nov. 1997.
44. P. Sastry, G. Santharam, and K. Unnikrishnan, "Memory neuron networks for identification and control of dynamical systems," IEEE Trans. Neural Networks, vol. 5, pp. 305-319, Mar. 1994.
45. A. Sperduti and A. Starita, "Supervised neural networks for the classification of structures, "IEEE Trans. Neural Networks, vol. 8, pp. 714-735, Mar. 1997.
46. C. Goller, "A connectionist control component for the theorem prover SETHEO," in Proc. ECAI'94 Workshop W14: Combining Symbolic and Connectionist Processing, 1994, pp. 88-93.
47. C. Nguyen and C. Widrow, "The truck backer-upper: An example of self learning in neural networks," in IEEE/INNS Int. Joint Conf. Neural Networks, vol. 1, Washington, D.C., 1989, pp. 357-364.
48. A. Atiya, S. El-Shoura, S. Shaheen, and M. El-Sherif, "A comparison between neural-network forecasting techniques - Case study: River flow forecasting," IEEE Trans. Neural Networks, vol. 10, pp. 402-409, Mar. 1999.
49. A. Bryson and Y.-C. Ho, Applied Optimal Control. Washington, D.C.: Hemisphere, 1975.
50. G. Golub and C. Van Loan, Matrix Computations. Baltimore, MD: Johns Hopkins Univ. Press, 1989.
51. A. Horn and C. Johnson, Matrix Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1985.
52. A. Weigend and N. Gerschenfeld, Eds., Time Series Prediction: Forecasting the Future and Understanding the Past. Reading, MA: Addison-Wesley, 1994.
53. L. Chan, "Training recurrent network with block-diagonal approximated Levenberg-Marquardt algorithm," in Proc. IJCNN'99, Washington, D.C., July 1999.