A second-order learning algorithm for multilayer networks based on block Hessian matrix

Neural Networks

Volumn 11, Issue 9, 1998, Pages 1607-1622

  Wang, Yi Jen ^a   Lin, Chin Teng ^a  

^a NATIONAL CHIAO TUNG UNIVERSITY   (Taiwan)

Author keywords

Forward backward propagation; Hessian matrix; Least squares estimation; Multilayer perceptrons; Newton's method

Indexed keywords

BACKPROPAGATION; COMPUTATIONAL METHODS; CONVERGENCE OF NUMERICAL METHODS; ERROR ANALYSIS; LEARNING ALGORITHMS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; MATRIX ALGEBRA; PROBLEM SOLVING; SENSITIVITY ANALYSIS; THEOREM PROVING;

BLOCK HESSIAN MATRIX; FORWARD-BACKWARD PROPAGATION SCHEMES; NEWTON'S METHOD; SECOND-ORDER LEARNING ALGORITHMS;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ANALYTIC METHOD; ANALYTICAL ERROR; ARTICLE; ARTIFICIAL NEURAL NETWORK; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; REGRESSION ANALYSIS; THEORY;

EID: 0032428930     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(98)00091-4     Document Type: Article

References (15)

1. Battiti R. First and second order methods for learning between steepest descent and Newton's method. Neural Computation. 4:1992;141-166.
2. Becker, S., & LeCun, Y. (1989). Improving the convergence of back-propagation learning with second order methods. In D. S. Touretzky, G. E. Hinton, & T. J. Sejnowski (Eds.), Proc. 1988 Connectionist Models Summer School. San Mateo, CA: Morgan Kaufmann, pp. 29-37.
3. Bello M. G. Enhanced training algorithms, and integrated training/architecture selection for multilayer perceptron networks. IEEE Transactions on Neural Networks. 3:1992;864-875.
4. Goodwin, G. C., & Sin, K. S. (1984). Adaptive Filtering Prediction and Control (chap. 3). Englewood Cliffs, NJ: Prentice-Hall.
5. Haykin, S. (1994). Neural Networks (chap. 7). New York: Macmillan.
6. Kailath, T. (1980). Linear Systems. Englewood Cliffs, NJ: Prentice-Hall.
7. LeCun, Y. (1989). Generalization and network design strategies. In Connectionism in Perspective. Amsterdam: North Holland, 143-155.
8. Lin, C. T., & Lee, C. S. G. (1996). Neural Fuzzy Systems: A Neural-Fuzzy Synergism to Intelligent Systems. Englewood Cliffs, NJ: Prentice-Hall.
9. Luenberger, D. G. (1984). Linear and Nonlinear Programming. Reading, MA: Addision-Wesley.
10. Makram-Ebeid, S., Sirat, J. A., & Viala, J. R. (1989). A rationalized back-propagation learning algorithm. In Proc. Int. Joint Conf. Neural Networks. Washington, DC, pp. 373-380. NJ: IEEE.
11. Piche S. W. Steepest descent algorithms for neural network controllers and filters. IEEE Transactions on Neural Networks. 5:1994;198-212.
12. Ricotti, L. P., Ragazzini, S., & Martinelli, G. (1988). Learning of word stress in a sub-optimal second order back-propagation neural network. In Proc. IEEE Int. Conf. Neural Networks. San Diego, CA, pp. 355-361. NJ: IEEE.
13. Silva, F., & Almeida, L. (1990). Acceleration techniques for the backpropagation algorithm. Lecture Notes in Computer Science, Vol. 412. Berlin: Springer, pp. 110-119.
14. Stefanos, K., & Anastassioy, D. (1988). Adaptive training of multilayer neural networks using a least-squares estimation technique. In Proc. IEEE Int. Conf. Neural Networks. San Diego, CA, pp. 383-390. NJ: IEEE.
15. Werbos P. J. Backpropagation through time: what it does and how to do it. Proc. of the IEEE. 78:1990;1550-1560.