Gradient Calculations for Dynamic Recurrent Neural Networks: A Survey

IEEE Transactions on Neural Networks

Volumn 6, Issue 5, 1995, Pages 1212-1228

  Pearlmutter, Barak A ^a  

^a SIEMENS CORPORATE RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; LEARNING SYSTEMS; VECTORS;

ELMAN HISTORY CUTOFF; FIXED POINT LEARNING ALGORITHM; FORWARD PROPAGATION; GRADIENT DESCENT METHOD; RECURRENT NEURAL NETWORKS;

NEURAL NETWORKS;

EID: 0029375851     PISSN: 10459227     EISSN: 19410093     Source Type: Journal    
DOI: 10.1109/72.410363     Document Type: Article

References (149)

1. J. L. McClelland, D. E. Rumelhart, and G. E. Hinton, “The appeal of parallel distributed processing.” in D. E. Rumelhart, J. L. McClelland, and the PDP research group, Eds., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations. MA: MIT 1986.
2. D. Marr and T. Poggio, “Cooperative computation of stereo disparity,” Sci., vol. 194, pp. 283-287, 1976.
3. G. Marr, D. Palm, and T. Poggio, “Analysis of a cooperative stereo Biol. vol. 28, 1978.
4. G. E. Hinton, “Relaxation and its role in vision,” Ph.D. dissertation, Univ. 1977.
5. L. S. Davis and A. Rosenfeld, “Cooperating processes for low-level vision: A Intell., vol. 1981.
6. R. Szeliski, “Cooperative algorithms for solving random-dot stereograms,” Carnegie Mellon Univ. Tech. Rep. CMU-CS-86-133, 1986.
7. G. E. Hinton, “Using relaxation to find a puppet,” in Proc A.I.S.B. Summer Univ. 1976.
8. D. L. Waltz and J. B. Pollack, “Massively parallel parsing: A strongly interactive model of natural language interpretation,” Cognitive Sci., vol. 9, pp. 51-74, 1985.
9. N. Qian and T. J. Sejnowski, “Learning to solve random-dot stereograms of dense and transparent surfaces with recurrent backpropagation,” in Proc. 1988 Connectionist Models Summer School, D. S. Touretzky, G. E. Hinton, and T. J. Eds., 1989, 435-443.
10. O. Nerrand, P. Roussel-Ragot, G. D. L. Personnaz, and S. Marcos, “Neural networks and nonlinear adaptive filtering: Unifying concepts and new algorithms,” Neural Computa., vol. 5, no. 2, pp. 165-197, 1993.
11. T. W. Karjala, D. M. Himmelblau. and R. Miikkulainen, “Data rectification using recurrent (Elman) neural networks,” in Proc. IJCNN '92, Baltimore, 901-905.
12. R. L. Watrous, B. Laedendorf, and G. M. Kuhn, “Complete gradient optimization of a recurrent network applied to BDG discrimination,” J Acoustical Soc. Amer., vol. 87, no. 3, 1301-1309, Mar. 1990.
13. P. Poddar and K. P. Unnikrishnan, “nonlinear prediction of speech signals using memory neuron networks,” in Proc. 1991 IEEE Workshop Neural Networks Process., IEEE Press, 1991, 395-404.
14. D. Albesano, R. Gemello, and F, Mana, “Word recognition with recurrent network automata,” in Proc. IJCNN '92, Baltimore, pp. 308-313.
15. S. R. Lockery, Y. Fang, and T. J. Sejnowski, “A dynamic neural network model of sensorimotor transformations in the leech,” Neural Computa., vol. 2, no. 3, 274-282, 1990.
16. K. Doya, M. E. T. Boyle, and A. I. Selverston, “Mapping between neural and physical activities of the lobster gastric mill system,” in Advances in Neural Information Processing Systems 5, S. J. Hanson, Jack D. Cowan, and C. Lee Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 913-920.
17. K. Doya, A. I. Selverston, and P. F. Rowat, “A Hodgkin-Huxley type neuron model that learns slow nonspike oscillation,” in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994.
18. Y. Chauvin and D. E. Rumelhart, Eds., Bcckpropagation: Theory, Architectures and Applications. Hillsdale, NJ: Lawrence Erlbaum Associates, 1994, in Press.
19. P. R. Gorman and T, J. Sejnowski, “Analysis of hidden units in a layered network trained to classify sonar targets,” Neural Networks, vol. 1, no. 1, pp. 75-89, 1988.
20. K. Lang and G. Hinton, “The development of the time-delay neural network architecture for speech recognition,” Dep. Comput. Sci., Carnegie Mellon Univ., Tech. Rep. CMU-CS-88-152, Nov. 1988.
21. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay networks,” IEEE Trans. Acoustics, Speech, Signal Process., vol. 37, no. 3, pp. 328-339, 1989.
22. K. J. Lang, G. E. Hinton, and A. Waibel, “A time-delay neural network architecture for isolated word recognition,” Neural Networks, vol. 3, no. 23-43, 1990.
23. K. J. Lang and G. E. Hinton, “Dimensionality reduction and prior knowledge in e-set recognition,” in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 178-185.
24. D. W. Tank and J. J. Hopfield, “Concentrating information in time: Analog neural networks with applications to speech recognition problems,” in Proc. IEEE 1st Int. Conf. Neural Networks, San Diego, CA, June 21-24 1987, 455—468.
25. S. P. Day and M. R. Davenport, “Continuous-time temporal backpropagation with adaptable time delays,” IEEE Trans. Neural Networks, vol. 4, no. 2, 348-354, 1993.
26. M. I. Jordan, “Generic constraints on underspecified target trajectories,” in Proc. IJCNN '89 (Int. Joint Conf. Neural Networks), Washington DC, June 18-22 1989.
27. Applications of Artificial Neural Networks, number 1294 in APIE Proceedings Series, Orlando, FL, Apr. 18-20, 1990.
28. A. S. Weigend, D. E. Rumelhart, and B. A. Huberman, “Generalization by weight-elimination with application to forecasting,” in Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, 875-882.
29. B. de Vries and J. C. Principe, “A theory for neural networks with time delays,” in Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: 1991. 162-168.
30. A. D. Back and A. C. Tsoi, “FIR and IIR synapses, a new neural network architecture for time series modeling,” Neural Computa., vol. 3, no. 3, pp. 337-350, 1991.
31. B. H. Juang, S. Y. Kung, and C. A. Camm, Eds., in Proc. 1991 IEEE Workshop Neural Networks Signal Process., IEEE Press, 1991.
32. D. Hush and B. Horne, “Progress in supervised neural networks,” IEEE Signal Process. Mag., vol. 10, no. 1, pp. 8-39, 1993.
33. B. de Vries and J. Principe, “The gamma model—A new neural network for temporal processing,” Neural Networks, vol. 5, no. 4, pp. 565-576, 1992.
34. T. Maxwell, C. L. Giles, Y. C. Lee, and H. H. Chen, “nonlinear dynamics of artificial neural systems,” in Proc. Snowbird Conf. Neural Networks Computing, Amer. Institute Physics, no. 151, 1986.
35. Y. Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541-551, 1989.
36. M. Kawato, T. Setoyama, and R. Suzuki, “Feedback error learning of movement by multi-layer neural networks,” in Proc. Int. Neural Networks Soc. 1st Annu. 1988, 342.
37. M. I. Jordan and R. A. Jacobs, “Learning to control an unstable system with forward modeling,” in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990.
38. K. S. Narendra and K. Parthasarathy, “Identification and control of dynamical systems using neural networks,” IEEE Trans. Neural Networks, vol. 1, pp. 4-27, Mar. 1990.
39. W. Thomas Miller, III, R. S. Sutton, and P. J. Werbos, Eds., Neural MA: MIT 1990.
40. H. T. Siegelmann and E. D. Sontag, “Turing computability with neural networks,” Appl. Math. Lett., vol. 4, no. 6, pp. 77-80, 1991.
41. J. Kilian and H. T. Siegelmann, “Computability with the classical sigmoid,” in Proc. 6th Annu. ACM Workshop Computational Learning Theory, Santa Cruz, CA, July 1993, pp. 137-143.
42. H. T. Siegelmann and E. D. Sontag, “Analog computation via neural networks,” in Proc. 2nd Israel Symp. Theory Comput. Syst., Natanya, Israel, June 1993.
43. A. Cleeremans, D. Servan-Schreiber, and J. McClelland, “Finite state automata and simple recurrent networks,” Neural Computa., vol. 1, no. 3, 372-381, 1989.
44. R. L. Watrous and G. M. Kuhn, “Induction of finite-state automata using second-order recurrent networks,” in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 309-316.
45. C. L. Giles, C. B. Miller, D. Chen, G. Z. Sun, H. H. Chen, and Y. C. Lee, “Extracting and learning an unknown grammar with recurrent neural networks,” in Advances in Neural Information Processing Systems 4J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 317-324.
46. M. C. Mozer and S. Das, “A connectionist symbol manipulator that discovers the structure of context-free languages,” in Advances in Neural Information Processing Systems 5S. J. Hanson, Jack D. Cowan, and C. Lee Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993,pp. 863-870.
47. S. Das, C. L. Giles, and G.-Z. Sun, “Using prior knowledge in a NNPDA to learn context-free languages,” in Advances in Neural Information Processing Systems 5S. J. Hanson, Jack D. Cowan, and C. Lee Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 65-72.
48. J. F. Kolen, “Fool's gold: Extracting finite state machines from recurrent network dynamics,” in Advances in Neural Information Processing Systems 6J. D. Cowan, G. Tesauro and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, In Press.
49. S. Das and M. C. Mozer, “A unified gradient-descent/clustering architecture for finite state machine induction,” in Advances in Neural Information Processing Systems 6J. D. Cowan, G. Tesauro and J. Eds. San Mateo, CA: Morgan Kaufmann, 1994, In Press.
50. D. Angluin, “Learning regular sets from queries and counter examples,” Inform. Computa., vol. 75, 87-106, 1987.
51. K. J. Lang, “Random DFA's can be approximately learned from sparse uniform examples,” in Proc. 5th Annu. ACM Workshop Computational Learning Theory, Pittsburgh, PA, July 1992, pp. 45-52.
52. J. Hochreiter, “Untersuchungen zu dynamischen neuronalen netzen,” 1991, Diplomarbeit, Institut für Informatik, Technische Universitat München.
53. M. C. Mozer, “Induction of multiscale temporal structure,” in Advances in Neural Information Processing Systems 4J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 275-282.
54. J. H. Schmidhuber, “Learning complex, extended sequences using the principle of history compression,” Neural Computa., vol. 4, no. 2, pp. 234-242, 1992.
55. “Learning unambiguous reduced sequence descriptions,” Advances in Neural Information Processing Systems 4in J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 291-298.
56. G. E. Hinton, “Learning distributed representations of concepts,” in Proc. 8th Annu. Cognitive Sci. Conf., 1986.
57. T. J. Sejnowski, P. K. Kienker, and G. Hinton, “Learning symmetry groups with hidden units: Beyond the perceptron,” Physica D, vol. 22, pp. 260-275, 1986.
58. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm for Boltzmann Machines,” Cognitive Sci., vol. 9, pp. 147-169, 1985.
59. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations D. E. Rumelhart, J. L. McClelland, and the PDP research group, Eds. Cambridge, MA: MIT Press, 1986.
60. P. J. Werbos, “Beyond regression: New tools for prediction and analysis in the behavioral sciences,” Ph.D. dissertation, Harvard Univ., 1974.
61. D. B. Parker, “Learning-logic,” MIT Center Res. Computational Economics Manage. Sci., Tech. Rep. TR-47, 1985.
62. R. Howard, Dynamic Programming and Karkhov Processes. Cambridge, MA: MIT Press, 1960.
63. D. S. Touretzky and D. A. Pomerleau, “What's hidden in the hidden layers?,” BYTE, pp. 227-233, Aug. 1989.
64. B. Widrow and M. Hoff, “Adaptive switching circuits,” in Proc. Western Electron. Show Conv., 1960, vol. 4, pp. 96-104, Institute of Radio Engineers (now IEEE).
65. Y. L. Cun, I. Kanter, and S. A. Solla, “Second-order properties of error surfaces: Learning time and generalization,” in R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds., Advances in Neural Information Processing Systems 3. San Mateo, CA: Morgan Kaufmann, 1991, pp. 918-924.
66. B. Baird, “A learning rule for CAM storage of continuous periodic sequences,” in Proc. IJCNN '90 II (Int. Joint Conf. Neural Networks), San Diego, CA, June 1990, pp. 493-498.
67. B. Baird and F. Eeckman, “CAM storage of analog patterns and continuous sequences with 3 n2 weights,” in Advances in Neural Information Processing Systems 3R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, 91-97.
68. C. A. Skarda and W. J. Freeman, “How brains make chaos to make sense of the world,” Brain Behavioral Sci., vol. 10, Nov. 1987.
69. W. J. Freeman, “Simulation of chaotic EEG patterns with a dynamic model of the olfactory system,” Biol. Cybern., vol. 56, p. 139, 1987.
70. J. P. Crutchfield and B. S. McNamara, “Equations of motion from a data series,” Complex Syst., vol. 1, pp. 417-452, 1987.
71. A. Lapedes and R. Farber, “nonlinear signal processing using neural networks: Prediction and system modeling,” Theoretical Division, Los Alamos Nat. Los Alamos, Tech. 1987.
72. F. Pineda, “Generalization of backpropagation to recurrent neural networks,” Physical Rev. Lett., vol. 19, no. 59, pp. 2229-2232, 1987.
73. L. B. Almeida, “A learning rule for asynchronous perceptrons with feedback in a combinatorial environment,” in Proc. IEEE 1st Int. Conf Neural Networks, San Diego, CA, June 21-24, 1987, pp. 609-618.
74. G. E. Hinton, “Deterministic Boltzmann learning performs steepest descent in weight-space,” Neural Computa., vol. 1, no. 1, pp. 143-150, 1989.
75. P. Baldi and F. Pineda, “Contrastive learning and neural oscillations,” Neural Computa., vol. 3, no. 4, pp. 526-545, 1991.
76. B. A. Pearlmutter, “Dynamic recurrent neural networks,” Carnegie Mellon Univ. School Comput. Sci., Pittsburgh, PA, Tech. Rep. CMUCS-90-196, Dec. 1990.
77. M. Gori, Y. Bengio, and R. de Mori, “BPS: A learning algorithm for capturing the dynamic nature of speech,” in Proc. IJCNN '89 Int. Joint Conf. Neural Networks, Washington DC, June 18-22 1989, pp. 417-423.
78. G. Kuhn, “A first look at phonetic discrimination using connectionist models with recurrent links,” Institute Defense Anal., Princeton, NJ, SCIMP Working Paper 82018, Apr. 1987.
79. M. C. Mozer, “A focused backpropagation algorithm for temporal pattern recognition,” Complex Syst., vol. 3, no. 4, pp. 349-381, Aug. 1989.
80. T. Uchiyama, K. Shimohara, and Y. Tokunaga, “A modified leaky integrator network for temporal pattern recognition,” in Proc. IJCNN '89 (Int. Joint Conf. Neural Networks), Washington DC, June 18-22 1989, pp. 469-475.
81. B. A. Pearlmutter, “Two new learning procedures for recurrent networks.” Neural Network Rev., vol. 3, no. 3, 99-101, 1990.
82. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Verrerling, Numerical Recipes in C. Cambridge, UK: Cambridge Univ. Press, 1988.
83. M. A. Cohen and S. Grossberg, “Stability of global pattern formation and parallel memory storage by competitive neural networks,” IEEE Trans. Syst., Man Cybern., vol. 13, pp. 815-826, 1983.
84. G. E. Hinton and T. J. Sejnowski, “Optimal perceptual inference,” in Proc. IEEE Conf. Comput. Vision Pattern Recognition, Washington DC, June 1983, pp. 448-453.
85. A. F. Atiya, “Learning on a general network,” in Neural Information Processing SystemsD. Z. Anderson, Ed. New York: American Institute of 1988 22-30.
86. S. Renals and R. Rohwer, “A study of network dynamics,” J. Statistical Physics, vol. 58. pp. 825-848, June 1990.
87. R. B. Allen and J. Alspector, “Learning of stable states in stochastic asymmetric networks,” Bell Commun. Res., Morristown, NJ, Tech. Rep. TM-ARH-015240, Nov. 1989.
88. C. C. Galland and G. E. Hinton, “Deterministic Boltzmann learning in networks with asymmetric connectivity,” Univ. Toronto Dep. Comput. Sci., Tech. 1989.
89. S. J. Nowlan, “Gain variation in recurrent error propagation networks,” Complex Syst., vol. 2, no. 3, 305-320, June 1988.
90. M. B. Ottaway, P. Y. Simard, and D. H. Ballard, “Fixed point analysis for recurrent neural networks,” in Advances in Neural Information Processing Systems I, David S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1989.
91. C. Peterson and J. R. Anderson, “A mean field theory learning algorithm for neural nets,” Complex Syst., vol. 1, 1987.
92. C. Peterson and J. R. Anderson, “A mean field theory learning algorithm for neural networks,” MCC (Microelectronics and Computer Technology Corporation), Austin, TX, tech. rep. EI-259-87, Aug. 1987.
93. G. E. Hinton and K. J. Lang, “Shape recognition and illusory conjunctions,” in Proc. 9th Int. Joint Conf. Artificial Intell., Los Angeles, Aug. 1985, vol. 1, pp. 252-259.
94. T. J. Sejnowski, “Higher-order Boltzmann machines,” in Proc. Snowbird Conf. Neural Networks Computing, Amer. Institute Physics, no. 151, 1986, pp. 398-403.
95. P. J. Werbos, “Backpropagation through time: What it does and how to do it,” in Proc. IEEE, vol. 78, pp. 1550-1560, 1990.
96. B. Pearlmutter, “Learning state space trajectories in recurrent neural networks,” Neural Computa., vol. 1, no. 2, pp. 263-269, 1989.
97. P. J. Werbos, “Generalization of backpropagation with application to a recurrent gas market model,” Neural Networks, vol. 1, pp. 339-356, 1988.
98. A. E. Bryson, Jr., “A steepest ascent method for solving optimum programming problems,” J. Appl. Mechanics, vol. 29, no. 2, p. 247, 1962.
99. S. E. Dreyfus, Dynamic Programming and the Calculus of Variations, vol. 21 of Mathematics in Science and Engineering. New York: Academic, 1965.
100. P. J. Werbos, “Applications of advances in nonlinear sensitivity analysis,” in Proc. 10th IFIP Conf Syst. Modeling Optimization, New York, Aug. 31-Sep. 4, 1981.
101. A. J. Robinson and F. Fallside, “Static and dynamic error propagation networks with application to speech coding,” in Proc. Int. Joint Conf. Neural Networks, Washington DC, June 18-22 1989, pp. 632-641.
102. M. Gherrity, “A learning algorithm for analog, fully recurrent neural networks,” in Proc. IJCNN '89 Int. Joint Conf. Neural Networks, Washington DC, June 18-22 1989, pp. 643-644.
103. R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Computa., vol. 1, no. 2, pp. 270-280, 1989.
104. K. S. Narendra and K. Parthasarathy, “Gradient methods for the optimization of dynamical systems containing neural networks,” IEEE Trans. Neural Networks, vol. no. Mar. 1991.
105. D. H. Jacobson, “New second-order and first-order algorithm for determining optimal control: A differential dynamic programming approach,” J. Optimization Theory Applicat., vol. 2, 1968.
106. R. E. Bellman, Methods of nonlinear Analysis: Volume II. New York: Academic, 1973.
107. A. Gelb et al., Eds., Applied Optimal Estimation. Cambridge, MA:
108. D. Zipser, “Subgrouping reduces complexity and speeds up learning in recurrent networks,” in Advances in Neural Information Processing Systems 2D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1990, pp. 638-641.
109. R. J. Williams and J. Peng, “An efficient gradient-based algorithm for on-line training of recurrent network trajectories,” Neural Computa., vol. 2, no. 4, 1990.
110. R. J. Williams and D. Zipser, “Gradient-based learning algorithms for recurrent networks and their computational complexity,” in Backpropagation: Theory, Architectures and Applications, Y. Chauvin and D. E. Rumelhart, Eds. Hillsdale, NJ: Lawrence Erlbaum Associates, 1994, in Press.
111. Jürgen H. Schmidhuber, “A fixed size storage O(n3) time complexity learning algorithm for fully recurrent continually running networks,” Neural Computa., vol. 4, no. 2, pp. 243-248, 1992.
112. G.-Z. Sun, H.-H. Chen, and Y.-C. Lee, “Green's function method for fast on-line learning algorithm of recurrent neural networks,” in Advances in Neural Information Processing Systems 4J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 333-340.
113. U. Bodenhausen, “Learning internal representations of pattern sequences in a neural network with adaptive time-delays,” in Proc. IJCNN '90 II (Int. Joint Conf. Neural Networks), San Diego, CA, June 1990.
114. D. W. Tank and J. J. Hopfield, “Neural computation by time compression,” Proc. National Academy Sci., vol. 84, pp. 1896-1900, 1987.
115. R. L. Watrous, “Speech recognition using connectionist networks,” Ph.D. dissertation, Univ. Pennsylvania, Oct. 1988.
116. P. Y. Simard, J. P. Rayzs, and B. Victorri, “Shaping the state space landscape in recurrent networks,” in Advances in Neural Information Processing Systems 3, R. P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, pp. 105-112.
117. J. L. Elman, “Finding structure in time,” Cognitive Sci., vol. 14, pp. 179-211, 1990.
118. “Finding structure in time,” Center Res. Language, Univ. Calif. San Tech. 1988.
119. Y. Le Cun, “Une procedure d'apprentissage pour reseau a seuil assy métrique,” in Cognitiva 85: A la Frontiére de I'Intelligence Artificielle des Sciences de la Connaissance des Neurosciences, Paris 1985, 1985, CESTA, Paris, 599-604.
120. T. Grossman, R. Meir, and E. Domany, “Learning by choice of internal representations,” Complex Syst., vol. 2, pp. 555-575, 1989.
121. R. Rohwer, “The ‘moving targets’ training algorithm,” in Advances in Neural Information Processing Systems 2, D. S. Touretzky, Ed. San Mateo, CA: Kaufmann, 1990, 558-565.
122. R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Univ. Calif San Diego, La CA Tech. ICS Nov. 1988.
123. M. I. Jordan, “Attractor dynamics and parallelism in a connectionist sequential machine,” in Proc. 9th Annu. Conf. Cognitive Sci. Soc., 1986, pp. 531-546.
124. M. B. Matthews, “Neural network nonlinear adaptive filtering using the extended Kalman filter algorithm,” in Proc. Int. Neural Networks Conf., Paris, July 1990, vol. 1, pp. 115-119.
125. R. J. Williams, “Training recurrent networks using the extended Kalman filter,” in Proc. IJCNN '92, (Int. Joint Conf. Neural Networks), Baltimore MD, 1992, 241-250.
126. R. E. Kalman, “A new approach to linear filtering and prediction problems,” Trans. ASME J. Basic Eng., vol. 82, no. 1, pp. 35-45, Mar. 1960.
127. R. K. Mahra, “On the identification of variances and adaptive Kalman filtering,” IEEE Trans. Automat. Contr., vol. AC-15, no. 2, pp. 175-184, Apr. 1970.
128. B. D. O. Anderson and J. B. Moore, Optimal Filtering. Englewood Cliffs, NJ: Prentice-Hall, 1979.
129. S. J. Nowlan and G. E. Hinton, “Adaptive soft weight tying using Gaussian mixtures,” in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, 993-1000.
130. J. Skilling, Ed., Maximum Entropy and Bayesian Methods. Norwell, MA: Kluwer, 1989.
131. P. F. Rowat and A. I. Selverston, “Learning algorithms for oscillatory networks with gap junctions and membrane currents,” Network: Computation Neural Syst., vol. 2, no. 1, 17-42, Feb. 1991.
132. J. G. Blom, J. M. Sanz-Serna, and J. G. Verwer, On Simple Moving Grid Methods for One-Dimensional Evolutionary Partial Differential Equations. Amsterdam: Stichting Mathematisch Centrum, 1986.
133. R. A. Jacobs, “Increased rates of convergence through learning rate adaptation,” Neural Networks, vol. 1, no. 4, pp. 295-307, 1988.
134. Y. Fang and T. J. Sejnowski, “Faster learning for dynamic recurrent backpropagation,” Neural Computa., vol. 2, no. 3, 270-273, 1990.
135. R. S. Sutton, “Gain adaptation beats least squares?,” in Proc. 7th Yale Workshop Adaptive Learning Syst., 1992, pp. 161-166.
136. “Adapting bias by gradient descent: An incremental version of delta-bar-delta,” in Proc. Nat. Conf Artificial Intell. AAAI-92, 1992.
137. M. A. Gluck, P. T. Glauthier, and R. S. Sutton, “Adaptation of cuespecific learning rates in network models of human category learning,” in Proc. 14th Annu. Sci. 1992.
138. D. J. Wilde and C. S. Beightler, Foundations of Optimization. Englewood Cliffs, NJ: Prentice-Hall, 1967.
139. B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson, Jr., “Stationary and nonstationary learning characteristics of the LMS adaptive filter,” Proc. IEEE, vol. 64, pp. 1151-1162, 1976.
140. B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985.
141. S. T. Alexander, Adaptive Signal Processing. New York: Springer-Verlag, 1986.
142. D. B. Parker, “Optimal algorithms for adaptive networks: Second-order backpropagation, second-order direct propagation and second-order Hebbian learning,” in Proc. IEEE 1st Int. Conf. Neural Networks, San Diego, CA, June 21-24, 1987, pp. 593-600.
143. R. Watrous, “Learning algorithms for connectionist networks: Applied gradient methods of nonlinear optimization,” in Proc. IEEE 1st Int. Conf Neural Networks, San Diego, CA, June 21-24, 1987, pp. 619-627.
144. J. J. Shynk and S. Roy, “The LMS algorithm with momentum updating,” in Proc. IEEE Int. Symp. Circuits Syst., June 6-9 1988, pp. 2651-2654.
145. G. Tesauro, Y. He, and S. Ahmad, “Asymptotic convergence of back- Neural vol. 1, no. 3, 382-391, 1989.
146. M. A. Tuğay and Y. Tanik, “Properties of the momentum LMS algorithm,” Signal Process., vol. 18, no. 2, 117-127, Oct. 1989.
147. B. A. Pearlmutter, “Gradient descent: Second-order momentum and saturating error,” in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992, pp. 887-894.
148. J. J. Hopfield and D. W. Tank, “‘Neural’ computation of decisions in optimization problems,” Biol. Cybern., vol. 52, pp. 141-152, 1985.
149. S. Kirkpatrick, C. D. Geiatt, Jr., and M. P. Vecchi, “Optimization by simulated annealing,” Sci., vol. 220, pp. 671-680, 1983.