Backpropagation algorithms for a broad class of dynamic networks

IEEE Transactions on Neural Networks

Volumn 18, Issue 1, 2007, Pages 14-27

  De Jesús, Orlando ^a   Hagan, Martin T ^b  

^a BAYLOR RESEARCH INSTITUTE   (United States)

^b Oklahoma State University   (United States)

Author keywords

Backpropagation through time (BPTT); Dynamic neural networks; Gradient; Jacobian; Layered digital dynamic network (LDDN); Real time recurrent learning (RTRL); Recurrent neural networks

Indexed keywords

ALGORITHMS; CALCULATIONS; RECURRENT NEURAL NETWORKS;

BACKPROPAGATION THROUGH TIME; DYNAMIC NEURAL NETWORKS; LAYERED DIGITAL DYNAMIC NETWORK; REAL-TIME RECURRENT LEARNING;

BACKPROPAGATION;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; CLUSTER ANALYSIS; COMPUTER ANALYSIS; METHODOLOGY; SIGNAL PROCESSING;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; CLUSTER ANALYSIS; COMPUTING METHODOLOGIES; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED; SIGNAL PROCESSING, COMPUTER-ASSISTED;

EID: 33846085516     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2006.882371     Document Type: Article

References (37)

1. A. F. Atiya and A. G. Parlos, "New results on recurrent network training: Unifying the algorithms and accelerating convergence," IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 697-709, May 2000.
2. N. E. Barabanov and D. V. Prokhorov, "Stability analysis of discrete-time recurrent neural networks," IEEE Trans. Neural Netw., vol. 13, no. 2, pp. 292-303, Mar. 2002.
3. P. Campolucci, A. Marchegiani, A. Uncini, and F. Piazza, "Signal-flow-graph derivation of on-line gradient learning algorithms," in Proc. Int. Conf. Neural Netw. (ICNN'97), Houston, TX, Jun. 9-12, 1997, pp. 1884-1889.
4. J. Choi, M. Bouchard, and T. H. Yeap, "Decision feedback recurrent neural equalization with fast convergence rate," IEEE Trans. Neural Netw., vol. 16, no. 3, pp. 699-708, May 2005.
5. O. De Jesús, "Training General Dynamic Neural Networks," Ph.D. dissertation, Oklahoma State Univ., Stillwater, OK, 2002.
6. O. De Jesús and M. Hagan, "Backpropagation through time for a general class of recurrent network," presented at the IEEE Int. Joint Conf. Neural Netw., Washington, DC, Jul. 2001.
7. O. De Jesús and M. Hagan, "Forward perturbation for a general class of dynamic network," in IEEE Int. Joint Conf. Neural Netw., Washington, DC, Jul. 2001, vol. 4, pp. 2626-2631.
8. H. B. Demuth and M. Beale, Users' Guide for the Neural Network Toolbox for MATLAB, ver. 4.0. Natick, MA: The Mathworks, 2000.
9. L. A. Feldkamp and G. V. Puskorius, "A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification," in Proc. IEEE, Nov. 1998, vol. 86, no. 11, pp. 2259-2277.
10. L. A. Feldkamp and D. V. Prokhorov, "Phased backpropagation: A hybrid of BPTT and temporal BP," in Proc. IEEE Int. Joint Conf. Neural Netw., 1998, vol. 3, pp. 2262-2267.
11. B. Fernandez, A. G. Parlos, and W. K. Tsai, "Nonlinear dynamic system identification using artificial neural networks (ANNs)," in Proc. Int. Joint Conf. Neural Netw., San Diego, CA, 1990, vol. 11, pp. 133-141.
12. M. Gupta, L. Jin, and N. Homma, Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory. New York: IEEE and Wiley, 2003.
13. M. Hagan, O. De Jesús, and R. Schultz, "Training recurrent networks for filtering and control," in Recurrent Neural Networks: Design and Applications, L. R. Medsker and L. C. Jain, Eds. Boca Raton, FL: CRC Press, 2000, pp. 325-354.
14. M. Hagan, H. B. Demuth, and M. Beale, Neural Network Design. Boston, MA: PWS, 1996.
15. M. Hagan and M. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.
16. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735-1780, 1997.
17. S. C. Kremer, "Spatio-temporal connectionist networks: A taxonomy and review," Neural Comput., vol. 13, no. 2, pp. 249-306, Feb. 2001.
18. J. R. Magnus and H. Neudecker, Matrix Differential Calculus. New York: Wiley, 1999.
19. D. Mandic and J. Chambers, Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. London, U.K.: Wiley, 2001.
20. K. S. Narendra and A. M. Parthasrathy, "Identification and control for dynamic systems using neural networks," IEEE Trans. Neural Netw., vol. 1, no. 1, p. 4, Jan. 1990.
21. G. V. Puskorius and L. A. Feldkamp, "Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 279-297, Mar. 1994.
22. D. Rumelhart, D. Hinton, and G. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, D. Rumelhart and F. McClelland, Eds. Cambridge, MA: MIT Press, 1986, vol. 1.
23. 3) time complexity learning algorithm for fully recurrent continually running networks," Neural Comput., vol. 4, pp. 243-248, 1992.
24. G. Z. Sun, H. H. Chen, and Y. C. Lee, "Green's method for fast on-line learning algorithm of recurrent neural networks," in In: Advances in Neural Information Processing Systems 4, J. Moody, S. Hanson, and R. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 333-340.
25. P. Tino, M. Cernansky, and L. Benuskova, "Markovian architectural bias of recurrent neural networks," IEEE Trans. Neural Netw., vol. 15, no. 1, pp. 6-15, Jan. 2004.
26. N. Toomarian and J. Barben, "Adjoint-functions and temporal learning algorithms in neural networks," in In: Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 113-120.
27. N. Toomarian and J. Barben, "Learning a trajectory using adjoint functions and teacher forcing," Neural Netw., vol. 5, pp. 473-484, 1992.
28. A. C. Tsoi and A. D. Back, "Locally recurrent globally feedforward networks: A critical review of architectures," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 229-239, Mar. 1994.
29. P. J. Werbos, "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences," Ph.D. dissertation, Harvard Univ., Cambridge, MA, 1974.
30. P. J. Werbos, "Backpropagation through time: What it is and how to do it," Proc. IEEE, vol. 78, no. 10, pp. 1550-1560, Oct. 1990.
31. R. J. Williams and J. Peng, "Gradient-based learning algorithms for recurrent neural networks and their computational complexity," in Backpropagation: Theory, Architectures, and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Hillsdale, NJ: Lawrence Erlbaum, 1992, pp. 433-486.
32. R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural Comput., vol. 1, pp. 270-280, 1989.
33. E. A. Wan, "Temporal backpropagation for FIR neural networks," in Proc. IEEE Int. Joint Conf. Neural Netw., 1990, vol. 1, pp. 575-580.
34. E. Wan and F. Beaufays, "Diagrammatic methods for deriving and relating temporal neural networks algorithms," in Adaptive Processing of Sequences and Data Structures, Lecture Notes in Artificial Intelligence, M. Gori and C. L. Giles, Eds. New York: Springer-Verlag, 1998.
35. W. Yang, "Neurocontrol Using Dynamic Learning," Ph.D. dissertation, Oklahoma State Univ., Stillwater, OK, 1994.
36. W. Yang and M. T. Hagan, "Training recurrent networks," in Proc. 7th Oklahoma Symp. Artif. Intell., Stillwater, OK, 1993, pp. 226-233.
37. P. J. Werbos, The Roots of Backpropagation. New York: Wiley, 1994.