Stable dynamic backpropagation learning in recurrent neural networks

IEEE Transactions on Neural Networks

Volumn 10, Issue 6, 1999, Pages 1321-1334

  Jin, Liang ^b   Gupta, Madan M ^a,c  

^a IEEE

^b NOKIA RESEARCH CENTER   (United States)

^c UNIVERSITY OF SASKATCHEWAN   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ADAPTIVE ALGORITHMS; BACKPROPAGATION; COMPUTER SIMULATION; CONSTRAINT THEORY; ERROR ANALYSIS; ITERATIVE METHODS; LYAPUNOV METHODS;

DYNAMIC BACKPROPAGATION (DBP) ALGORITHMS; DYNAMIC NEURAL NETWORKS;

RECURRENT NEURAL NETWORKS;

EID: 0033352722     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.809078     Document Type: Article

References (43)

1. F. J. Pineda, "Generalization of backpropagation to recurrent neural networks," Phys. Rev. Lett., vol. 59, no. 19, pp. 2229-2232, 1987.
2. F. J. Pineda, "Dynamics and architecture for neural computation," J. Complexity, vol. 4, pp. 216-245, 1988.
3. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," in Proc. Nat. Academy Sci. USA, vol. 79, 1982, pp. 2554-2558.
4. J. Hopfield, "Neurons with graded response have collective computational properties like those of two state neurons," in Proc. Nat. Academy Sci. USA, vol. 81, 1984, pp. 3088-3092.
5. L. B. Almeida, "A learning rule for asynchronous perceptrons with feedback in a combinatorial environment," in Proc. IEEE 1st Conf. Neural Networks, San Diego, CA, June 21-24, 1987, vol. II, pp. 609-618.
6. S. Amari, "Learning patterns and pattern sequences by self-organizing nets of threshold elements," IEEE Trans. Comput., vol. c-21, pp. 1197-1206, Nov. 1972.
7. S. Amari, "Neural theory of association and concept formation," Biol. Cybern., vol. 26, pp. 175-185, 1977.
8. M. A. Arbib, Brains, Machines, and Mathematics. New York: McGraw-Hill, 1988.
9. T. Kohonen, Associative Memory: A System Theoretical Approach. New York: Springer-Verlag, 1977.
10. B. Kosko, "Bidirectional associative memories," IEEE Trans. Syst., Man, Cybern., vol. SMC-18, pp. 42-60, 1988.
11. Y. Chauvin, "Dynamic behavior of constrained backpropagation networks," in Advances in Neural Information Processing System, D. A. Touretzky, Ed., vol. 2. San Mateo, CA: Morgan-Kaufmann, pp. 519-526, 1990.
12. A. Guez, V. Protopopsecu, and J. Bahren, "On the stability, storage capacity, and design of continuous nonlinear neural networks," IEEE Trans. Syst., Man, Cybern., vol. 18, pp. 80-87, Jan./Feb. 1988.
13. S. I. Sudharsanan and M. K. Sunareshan, "Equilibrium characterization of dynamical neural networks and a systematic synthesis procedure for associative memories," IEEE Trans. Neural Networks, vol. 2, pp. 509-521, Sept. 1991.
14. R. Kamimura, "Activated hidden connections to accelerate the learning in recurrent neural networks," in Proc. Int. Joint Conf. Neural Networks (IJCNN), 1992, pp. I-693-700.
15. R. Tawel, "Nonlinear functional approximation with networks using adaptive neurons," in Proc. Int. Joint Conf. Neural Networks (IJCNN), 1992, pp. III-491-496.
16. A. Atiya and Y. S. Abu-Mostafa, "An analog feedback associative memory," IEEE Trans. Neural Networks, vol. 4, no. 1, pp. 117-126, Jan. 1993.
17. Y. S. Abu-Mostafa and J.-M. St. Jacques, "Information capacity of the Hopfield model," IEEE Trans. Inform. Theory, vol. IT-31, pp. 461-464, 1984.
18. M. Morita, "Associative memory with nonmonotone dynamics," Neural Networks, vol. 6, no. 1, pp. 115-126, 1993.
19. A. H. Gee, S. V. B. Aiyer, and R. W. Prager, "An analytical framework for optimizing neural networks," Neural Networks, Vol. 6, no. 1, pp. 79-98, 1993.
20. S. V. B. Aiyer, M. Niranjan, and F. Fallside, "A theoretical investigation into the performance of the Hopfield model," IEEE Trans. Neural Networks, vol. 1, pp. 204-215, 1990.
21. J. Farrell and A. Michel, "A synthesis procedure for Hopfield's continuous-time associative memory," IEEE Trans. Circuits Syst., vol. 37, pp. 877-884, 1990.
22. A. Michel and J. Farrell, "Associative memories via artificial neural networks," IEEE Contr. Syst. Mag., pp. 6-17, Apr. 1990.
23. S. Grossberg, "Nonlinear neural networks: Principles, mechanisms and architectures," Neural Networks, vol. 1, no. 1, pp. 17-61, 1988.
24. M. A. Cohen and S. Grossberg, "Absolute stability of global pattern information and parallel memory storage by competitive neural networks," IEEE Trans. Syst., Man, Cybern., vol. SMC-13, pp. 815-826, 1983.
25. A. Guez, V. Protopopsecu, and J. Barhen, "On the stability, storage capacity, and design of nonlinear continuous neural networks," IEEE Trans. Syst., Man, Cybern., vol. SMC-18, pp. 80-87, 1988.
26. D. G. Kelly, "Stability in contractive nonlinear neural networks," IEEE Trans. Biomed. Eng., vol. 37, pp. 231-242, 1990.
27. J. A. Anderson, J. W. Silverstein, S. A. Ritz, and R. S. Jones, "Distinctive features, categorical perception, and probability learning: Some applications of a neural model," Neurocomputing: Foundations of Research, J. A. Anderson and E. Rosenfeld, Eds. Cambridge, MA: MIT Press, 1988.
28. A. H. Michel, J. Si, and G. Yen, "Analysis and synthesis of a class of discrete-time neural networks described on hypercubes," IEEE Trans. Neural Networks, vol. 2, pp. 32-46, 1991.
29. L. K. Li, "Fixed point analysis for discrete-time recurrent neural networks," in Proc. IJCNN, June 1992, vol. IV, pp. 134-139.
30. L. Jin, P. N. Nikiforuk, and M. M. Gupta, "Absolute stability conditions for discrete-time recurrent neural networks," IEEE Trans. Neural Networks, vol. 5, pp. 954-964, 1994.
31. L. Jin and M. M. Gupta, "Globally asymptotical stability of discrete-time analog neural networks," IEEE Trans. Neural Networks, vol. 7, pp. 1024-1031, 1996.
32. L. Jin and M. M. Gupta, "Equilibrium capacity of analog feedback neural networks," IEEE Trans. Neural Networks, vol. 7, pp. 782-787, 1996.
33. E. K. Blum and X. Wang, "Stability of fixed points and periodic orbits and bifurcations in analog neural networks," Neural Networks, vol. 5, no. 4, pp. 577-587, 1992.
34. C. M. Marcus and R. M. Westervelt, "Dynamics of iterated map neural networks," Phys. Rev. A, Vol. 40, no. 1, pp. 577-587, 1989.
35. C. M. Marcus, F. R. Waugh, and R. M. Westervelt, "Associative memory in an analog iterated-map neural network," Phys. Rev. A, vol. 41, no. 6, pp. 3355-3364, 1990.
36. D. E. Rumelhart and J. L. McCelland, "Learning internal representations by error propagation," Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1. Cambridge, MA: MIT Press, 1986.
37. R. Hecht-Nielsen, "Theory of the backpropagation neural network," in Proc. Int. Joint Conf. Neural Networks, June 1989, pp. I-593-605.
38. K. S. Narendra and K. Parthasarathy, "Identification and control of dynamical systems using neural networks," IEEE Trans. Neural Networks, vol. NN, pp. 4-27, Mar. 1990.
39. K. S. Narendra and K. Parthasarthy, "Gradient methods for the optimization of dynamical systems containing neural networks," IEEE Trans. Neural Networks, vol. 2, pp. 4-27, 1991.
40. K. J. Hunt, D. Sbarbaro, R. Zbikowski, and P. J. Gawthrop, "Neural networks for control systems - A survey," Automatica, vol. 28, no. 6, pp. 1083-1112, 1992.
41. D. R. Hush and B. G. Horne, "Progress in supervised neural networks: What's new since Lippmann?," IEEE Signal Processing Mag., no. 1, pp. 8-39, Jan. 1993.
42. A. E. Bryson and Y. C. Ho, Applied Optimal Control. New York: Blaisdell, 1969.
43. R. A. Horn and C. A. Johnson, Matrix Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1985.