On structure-exploiting trust-region regularized nonlinear least squares algorithms for neural-network learning

Neural Networks

Volumn 16, Issue 5-6, 2003, Pages 745-753

  Mizutani, Eiji ^a   Demmel, James W ^b  

^a NATIONAL TSING HUA UNIVERSITY   (Taiwan)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Block angular and block arrow matrices; Krylov subspace; Neuro fuzzy modular networks; Trust region methods

Indexed keywords

COMPUTER SIMULATION; EXPERT SYSTEMS; FUZZY SETS; LEARNING ALGORITHMS; LEAST SQUARES APPROXIMATIONS; REGRESSION ANALYSIS;

NEURO-FUZZY MODULAR NETWORKS;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ARTIFICIAL NEURAL NETWORK; CONFERENCE PAPER; FISHER EXACT TEST; MATHEMATICAL MODEL; NONLINEAR SYSTEM; NORMAL DISTRIBUTION; PRIORITY JOURNAL; REGRESSION ANALYSIS;

EID: 0038383454     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0893-6080(03)00085-6     Document Type: Conference Paper

References (21)

1. Amari S.-I. Natural gradient works efficiently in learning. Neural Computation. 10:1998;251-276.
2. Björck A. Numerical methods for least squares problems. 1996;SIAM.
3. Conn A.R., Gould N.I.M., Toint P.L. MPS/SIAM series on optimization. MPS/SIAM series on optimization. 2000;SIAM.
4. Demmel J. Applied numerical linear algebra. 1997;SIAM.
5. Dennis J.E., Gay D.M., Welsch R.E. An adaptive nonlinear least-squares algorithm. ACM Transactions on Mathematical Software. 7:(3):1981;348-368.
6. Golub, G. H., & LeVeque, R. J (1978). Extensions and uses of the variable projection algorithm for solving nonlinear least squares problems. Technical Report SU 326, Department of computer Science, Stanford University (also ARO Report 79-3).
7. Hastie T., Tibshirani R., Friedman J. The elements of statistical learning. 2002;Springer, Berlin.
8. Kaufman L., Sylvester G. Separable nonlinear least squares with multiple right-hand sides. SIAM Journal on Matrix Analysis and Applications. 13:(1):1992;68-89.
9. Masters T. Advanced algorithms for neural networks: a C++ sourcebook. 1995;Wiley, New York.
10. Mizutani E. Powell's dogleg trust-region steps with the quasi-Newton augmented Hessian for neural nonlinear least-squares learning. Proceedings of the IEEE International Conference on Neural Networks, Washington, DC. 2:1999;1239-1244.
11. Mizutani E., Dreyfus S.E., Nishio K. On derivation of MLP backpropagation from the Kelley-Bryson optimal-control gradient formula and its application. Proceedings of the IEEE International Conference on Neural Networks, Como, Italy. 2:2000;167-172.
12. Mizutani E., Demmel J.W. On iterative Krylov-dogleg trust-region steps for solving neural networks nonlinear least squares problems. Leen T., Dietterich T., Tresp V. Advances in neural information processing systems, Vol. 13 (NIPS 2000). 2000;605-611 MIT Press, Cambridge, MA.
13. Mizutani E., Dreyfus S.E. On complexity analysis of supervised MLP-learning for algorithmic comparisons. Proceedings of the INNS-IEEE International Joint Conference on Neural Networks, Washington, DC. 1:2001;347-352. Available at http://www.ieor.berkeley.edu/people/faculty/dreyfus-pubs/IJCNN01.ps.
14. Mizutani E., Demmel J.W. On separable nonlinear least squares algorithms for neuro-fuzzy modular network learning. Proceedings of the International Joint Conference on Neural Networks, Honolulu USA. 2002;2399-2404.
15. Mizutani E., Nishio K. Fuzzy mixtures of complementary local experts: towards neuro-fuzzy modular networks. Proceedings of the IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Honolulu USA. 2:2002;1192-1197.
16. Mizutani, E., & Demmel, J.W. (2003a). On augmenting neural-network learning capacity for large-scale classification benchmarks by iterative batch/block-mode trust-region Krylov-dogleg algorithms. To appear in Proceedings of the Third Int'l DCDIS Conf. on Engineering Applications and Computational Algorithms. Guelph CANADA, May.
17. Mizutani, E., & Demmel, J.W. (2003b). Iterative scaled trust-region learning in Krylov subspaces via Pearlmutter's implicit spares Hessian-vector multiply. (Working manuscript).
18. Moré J.J. Recent developments in algorithms and software for trust region methods. Bachem A., Grotschel M., Korte B. Mathematical programming: the state of the art. 1983;258-287 Springer, Berlin.
19. Nocedal J., Wright S.J. Springer series in operations. Springer series in operations. 1999;Springer, Berlin.
20. Pearlmutter B.A. Fast exact multiplication by the Hessian. Neural Computation. 6:(1):1994;147-160.
21. Toint P.L. On Large scale nonlinear least squares calculations. SIAM Journal on Scientific Computing. 8:(3):1987;416-435.