Dynamic recurrent neural networks: A dynamical analysis

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 26, Issue 5, 1996, Pages 692-706

  Draye, Jean Philippe S ^a,b   Pavisic, Davor A ^a,c   Cheron, Guy A ^d,e   Libert, Gaétan A ^a  

^a FACULTÉ POLYTECHNIQUE DE MONS   (Belgium)

^b FUND FOR SCIENTIFIC RESEARCH   (Belgium)

^c Universitad del Valle ^*

^d UNIVERSITY OF MONS   (Belgium)

^e UNIVERSITÉ LIBRE DE BRUXELLES   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; LEARNING SYSTEMS; PROBABILITY; STATISTICAL METHODS; SYSTEM STABILITY;

ADAPTATIVE PARAMETER; ADAPTATIVE TIME CONSTANT; DYNAMIC RECURRENT NEURAL NETWORKS; NEURAL TRANSFORMATION;

NEURAL NETWORKS;

EID: 0030271093     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/3477.537312     Document Type: Article

References (76)

1. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, "A learning algorithm for Boltzmann machines," Cognitive Science, vol. 2, pp. 147-169, 1985.
2. L. B. Almeida, "A learning rule for asynchronous perceptrons with feedback in a combinatorial environment," M. Caudill and C. Butler, Eds, in Proc. IEEE Int. Conf. Neural Networks, , IEEE, New York, 1987, vol. 2, pp. 609-618.
3. _, "Backpropagation in perceptrons with feedback," in Neural Computers, R. Eckmiller and C. V. D. Malsburg, Eds. New York: Springer-Verlag, 1988.
4. _, "Backpropagation in nonfeedforward networks," in Neural Computing Architectures, I. Aleksander Ed. Cambridge, MA: M.I.T. Press, 1989.
5. S. I. Amari, "A method of statistical neurodynamics," Kybernetik, vol. 14, pp. 201-215, 1974.
6. _, "Mathematical foundations of neurocomputing," in Proc. IEEE, vol. 78, pp. 1443-1463, 1990,
7. S. I. Amari, K. Yoshida, and K. Kanatani, "A mathematical foundations for statistical neurodynamics," SIAM J. Appl. Math, vol. 33, pp. 95-126, 1977.
8. K. L. Babcock and R. M. Westervelt, "Dynamics of simple electronics neural networks," Physica, vol. 28D, pp. 305-315, 1987.
9. Y. Bengio, P. Simard, and P. Frasconi, "Learning long-term dependencies with gradient descent is difficult," IEEE Trans. Neural Networks, vol. 5, no. 2, pp. 157-166, 1994.
10. E. K. Blum, Numerical Analysis and Computation: Theory and Practice. Reading, MA: Addison-Wesley, 1972.
11. E. K. Blum and X. Wang, "Stability of fixed points and periodic points and bifurcations in analog neural networks," IEEE Trans. Neural Networks, vol. 5, pp. 577-587, 1994.
12. V. G. Boltyanskii, R. V. Gamkrelidze, E. F. Mishchenko, and L. S. Pontryagin, "The maximum principle in the theory of optimal processes of control," in Optimal and Self-Organizing Control, R. Oldenburger, Ed. Cambridge, MA: MIT Press, 1966, pp. 262-266.
13. G. Cheron, E. Godaux, J. M. Laune, and B. Vanderkelen, "Lesions in the cat prepositus complex: Effects on the vestibulo-ocular reflex and saccades," J. Physiology (London), vol. 372, pp. 75-94, 1986.
14. H. D. Chiang and J. S. Thorp, "Stability regions of nonlinear dynamical systems: A constructive methodology," IEEE Trans. Automat. Contr., vol. 34, no. 12, pp. 1229-1241, 1989.
15. K. Ciliz and A. Harova, "Stability regions of recurrent type neural networks," Electron. Lett., vol. 28, no. 11, pp. 1022-1024, 1992.
16. K. Doya, "Learning temporal patterns in recurrent neural networks," in Proc. IEEE Syst., Man, Cybern. Conf., 1990, pp. 170-172.
17. _, "Universality of fully-connected recurrent neural networks," University of California, San Diego, Tech. Rep., 1993.
18. K. Doya and S. Yoshizawa, "Adaptative neural oscillator using continuous-time backpropagation learning," Neural Networks, vol. 2, pp. 375-386, 1989.
19. _, "Memorizing oscillatory patterns in the analog neuron network," in Proc. Int. Joint Conf. Neural Networks, IEEE, Washington D.C., 1989.
20. J. P. Draye, G. Cheron, M. Bourgeois, and G. Libert, "A dynamic neural network identification of electromyography and wrist position relationship during complex movements," Society for Neuroscience Abstracts, vol. 20, part I, p. 853, 1994.
21. J. P. Draye, G. Cheron, M. Bourgeois, D. Pavisic, and G. Libert, "Identification of the human arm kinetics using dynamic recurrent neural networks," Michel Verleysen, Ed., in Proc. European Symp. Artificial Neural Networks (ESANN'95), DFacto, Brussels, 1995, pp. 33-38.
22. J. P. Draye, G. Cheron, G. Libert, and E. Godaux, "Supervised neural network simulation of the integrator of the oculomotor system," Journal of Biomechanics, vol. 27, no. 6, p. 775, 1994.
23. J. P. Draye and G. Libert, "Dynamic recurrent neural networks: Theoretical aspects and optimization," Neural Network World, vol. 3, no. 6, pp. 705-714, 1993.
24. R. C. Eberhart, "Standardization of neural network terminology," IEEE Trans. Neural Networks, vol. 1, no. 2, pp. 244-245, 1990.
25. J. L. Elman, "Finding structure in time," Cognitive Science, vol. 14, pp. 179-211, 1990.
26. E. Fiesler and H. J. Caulfield, "Neural network formalization," Institut Del Molle d'Intelligence Artificielle (IDIAP), Suisse-Alabama A&M University, AL, Tech. Rep., 1991.
27. M. Garzon and S. Franklin, "Global dynamics in neural networks," Complex Systems, vol. 3, pp. 29-36, 1989.
28. M. Gherrity, "A learning algorithm for fully analog recurrent fully recurrent neural networks," in Proc. Int. Joint Conf. Neural Networks, IEEE, 1989, vol. 2, pp. 643-644.
29. C. L. Giles, "Learning and extracting finite state automata with second-order recurrent neural networks," Neural Computation, vol. 4, pp. 393-405, 1992.
30. P. E. Gill, W. Murray, and M. H. Wright, Practical Optimization. New York: Academic, 1981.
31. J. Guckenheimer and P. J. Holmes, "Non-linear oscillations, dynamical systems and bifurcation of vector fields," in Applied Mathematical Science, vol. 42. New-York/Berlin: Springer-Verlag, 1983.
32. J. K. Hale, Ordinary Differential Equations. New York: Krieger, 1980.
33. M. W. Hirsch, "Convergent activation dynamics in continuous time networks," Neural Networks, vol. 2, no. 5, pp. 331-349, 1989.
34. _, "Saturation at high gain in discrete time recurrent networks," Neural Networks, vol. 7, no. 3, pp. 449-453, 1994.
35. J. H. Holland, "Properties of the bucket brigade," in Proc. Int. Conf. Genetic Algorithms, Hillsdale, NJ, 1985.
36. J. J. Hopfield, "Neural networks as physical systems with emergent collective computational abilities," in Proc. National Academy of Sciences, 1982, vol. 79, pp. 2554-2558.
37. _, "Neurons with graded response have collective computational properties like those of two-state neurons," in Proc. National Academy of Science USA, 1984, vol. 81.
38. J. J. Hopfield and D. W. Tank, "Neural computation of decisions in optimization problems," Biological Cybernetics, vol. 52, 1985.
39. K. Hornik, M. Stinhcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, pp. 359-366, 1989.
40. M. I. Jordan, "Attractor dynamics and parallelism in a connectionist sequential machine," in Proc. 8th Annu. Conf. Cognitive Science Society, 1989, vol. 79, pp. 531-546.
41. S. W. Kuffler, J. G. Nicholis, and A. R. Martin, From Neuron to Brain, 2nd ed. Sunderlands, MA: Sinauer Associates Inc., 1984.
42. G. Kung, "Connected recognition with a recurrent network," Speech Communication, vol. 9, no. 2, 1990.
43. A. Lapedes and R. Farber, "A self-optimizing, nonsymmetrical neural net for content addressable memory and pattern recognition," Physica D, vol. 22, pp. 247-259, 1986.
44. J. P. LaSalle, "The stability of dynamical systems," in SIAM Regional Conferences Series in Applied Mathematics, SIAM, vol. 25, 1976.
45. J. P. LaSalle and S. Lefschetz, Stability by Lyapunov's Direct Method. New York: Academic, 1961.
46. C. M. Marcus and R. M. Westervelt, "Dynamics of iterated-map neural networks," Phys. Rev., vol. A40, no. 1, pp. 501-504, 1989.
47. _, "Stability of analog neural networks with delay," Phys. Rev., vol. A39, no. 1, pp. 347-359, 1989.
48. _, "Associative memory in an analog iterated-map neural networks," Phys. Rev., vol. A41, no. 6, pp. 3355-3364, 1990.
49. A. N. Michel, J. A. Farrel, and W. Porod, "Qualitative analysis of neural networks," IEEE Trans. Circuits Syst., vol. 36, no. 2, pp. 229-243, 1989.
50. A. N. Michel, N. R. Sarabudla, and R. K. Miller, "Stability analysis of complex dynamical systems: Some computational methods," Circuits, Syst. Signal Proc., vol. 1, no. 2, pp. 171-202, 1982.
51. M. C. Mozer, "Neural net architectures for temporal sequence processing," in Predicting the Future and Understanding the Past: A Comparison of Approaches, A. S. Weigend and N. A. Gershenfeld Eds. Reading, MA: Addison-Wesley, 1993.
52. P. Naslin, Théorie de la Commande et Conduite Optimale, Bibliothèque de l'Automaticien. Paris: Dunod, 1969.
53. S. J. Nowlan, "Gain variation in recurrent error propagation networks," Complex Systems, vol. 2, 1988.
54. B. A. Pearlmutter, "Learning state space trajectories in recurrent neural networks," Neural Computat., vol. 1, pp. 263-269, 1989.
55. F. Pineda, "Generalization of backpropagation to recurrent neural networks," Phys. Rev. Lett., vol. 19, pp. 2229-2232, 1987.
56. _, "Dynamics and architecture for neural computation," Journal of Complexity, vol. 1, pp. 263-269, 1988.
57. _, "Generalization of backpropagation to recurrent and higher order neural networks," Neural Information Processing Systems, D. Anderson, Ed. Morgan-Kaufmann Pub., 1988.
58. R. Rohwer, "Training time-dependence in neural networks," in Proc. 1st Int. Conf. Neural Networks, IEEE, pp. 701-708, 1987.
59. S. Renals and R. Rohwer, "A study of network dynamics," J. Statistical Physics, vol. 58, no. 5, pp. 825-848, 1990.
60. G. Riedel, "Temporal sequences and chaos in neural networks," Phys. Rev. A, vol. 36, p. 1428, 1988.
61. A. J. Robinson and F. Fallside, "The utility driven dynamic error propagation network," Engineering Department, Cambridge Univ., Cambridge, England, Techn. Rep. CUED/F-INFENG/TR.1, 1987.
62. P. F. Rowat and A. I. Selverston, "Learning algorithms for oscillatory networks with gap junction and membrane currents," Networks, vol. 2, pp. 17-41, 1991.
63. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," Parallel Distributed Processing: Explorations of the Microstructure of Cognition. Cambridge MA: Bradford, vol. 1, 1986.
64. W. Schiffman, M. Joost, and R. Werner, "Optimization of the backpropagation algorithm for training multilayer perceptrons," Institute of Physics, Univ. of Koblenz, Germany, Tech. Rep. 1992.
65. J. Schmidhuber, "A local learning algorithm for dynamic feedforward and recurrent networks," Connection Sci., vol. 1, pp. 403-42, 1989.
66. H. T. Siegelmann and E. D. Sontag, "On the computational power of neural nets," in Proc. 5th ACM Workshop Computational Learning Theory (COLT'92), 1992.
67. F. M. Silva and L. B. Almeida, "Speeding up backpropagation," in Advanced Neural Computers, R. Eckmiller, Ed. Amsterdam, The Netherlands: Elsevier, 1990, pp. 151-158.
68. P. Y. Simard, "Analysis of recurrent back-propagation," in Proc. Connectionist Models Summer Schools, 1988.
69. W. S. Stornetta, T. Hogg, and B. A. Huberman, "A dynamical approach to temporal pattern processing," in Proc. IEEE Conf. Neural Information Processing Systems, 1987, pp. 750-759.
70. R. S. Sutton, "Learning to predict by the methods of temporal difference," Machine Learning, vol. 3, pp. 9-44, 1988.
71. S. Thrun and F. Smieja, "A general feed-forward algorithm for gradient-descent in connectionist networks," GMD-German National Research Center for Computer Science, Tech. Rep. Nr TR 483, Nov. 1990.
72. M. Vidyasagar, "Location and stability of the high-gain equilibria of nonlinear neural networks," IEEE Trans. Neural Networks, vol. 4, no. 4, pp. 425-442, 1993.
73. X. Wang and E. K. Blum, "Discrete-time versus continuous-time models of neural networks," in Journal of Computer and System Sciences. New York: Academic, 1992, vol. 45, pp. 1-19.
74. A. Widrow and M. Hoff, "Adaptative switching circuits," in Western Electronic Show and Convention, Convention Record, Institute of Radio Engineers, vol. 4, pp. 96-104, 1960.
75. R. I. Williams and D. Zipser, "A Learning algorithm for continually running fully recurrent neural networks," Neural Computat., vol. 1, pp. 270-280, 1989.
76. G. Cheron, J. P. Draye, M. Bourgeois, and G. Libert, "A dynamic neural network identification of electromyography and arm trajectory relationship during complex movements," IEEE Trans. Biomed. Eng., vol. 43, no. 5, pp. 552-558, 1996.