Training multilayer neural networks using fast global learning algorithm - Least-squares and penalized optimization methods

Neurocomputing

Volumn 25, Issue 1-3, 1999, Pages 115-131

  Cho, Siu Yeung ^a   Chow, Tommy W S ^a  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Global learning algorithm; Least squares method; Multilayer neural networks; Penalized optimization

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; ERROR ANALYSIS; HEURISTIC METHODS; LEARNING ALGORITHMS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; OPTIMIZATION; PROBLEM SOLVING;

PENALIZED OPTIMIZATION;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; MATHEMATICAL ANALYSIS; MATHEMATICAL COMPUTING; PRIORITY JOURNAL; PROBLEM SOLVING; REGRESSION ANALYSIS; SIMULATION;

EID: 0032935466     PISSN: 09252312     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0925-2312(99)00055-7     Document Type: Article

References (32)

1. Aarts E., Korst J. Simulated Annealing and Boltzmann Machines. A Stochastic Approach to Combinatorial Optimization and Neural Computing:1989;Wiley, New York.
2. M.R. Azimi-Sadjadi, S. Citrin, Fast learning process of multilayer neural nets using recursive least squares technique, Proceedings of IEEE International Conference on Neural Networks, Washington DC, May 1989.
3. Azimi-Sadjadi M.R., Liou R. Fast learning process of multilayer neural networks using recursive least squares technique. IEEE Trans. Signal Process. 40(2):1992;446-450.
4. Battiti R. First- and second-order methods for learning: between steepest descent and Newton's method. Neural Computa. 4(2):1992;141-166.
5. Battiti R., Tecchiolli G. Training neural nets with the reactive tabu search. IEEE Trans. Neural Networks. 6(5):1995;1185-1200.
6. Cho S.-Y., Chow T.W.S. An efficient and stable recurrent neural networks training algorithm for time series forecasting. Int. J. Knowledge-Based Intell. Eng. Systems. 1(3):1997;138-148.
7. Chow T.W.S., Cho S.Y. An accelerated recurrent network training algorithm using IIR filter model and recursive least squares method, IEEE Trans. Circuits Systems - I. Fund Theory Appl. 44(11):1997;1082-1086.
8. N. Dodd, Optimization of network structure using genetic techniques, International Joint Conference on Neural Networks, San Diego, 1990.
9. Duda R.O., Hart P.E. Pattern Classification and Scene Analysis. 1973;Wiley, New York.
10. S.E. Fahlman, C. Lebiere, The cascade-correlation learning architecture, Technical Report, CMU-CS-90-100, School of Computer Science, Carnegie Mellon University, 1990.
11. Floudas C.A., Pardalos P.M. State of the Art in Global Optimization. 1996;Kluwer Academic Publishers, Dordrecht, The Netherlands.
12. Goffe W.L., Ferrier G.D., Rogers J. Global optimization of statistical functions with simulated annealing. J. Econometrics. 60:1994;65-99.
13. M. Gori, A. Tesi, Some examples of local minima during learning with back-propagation, Parallel Architectures and Neural Networks, Vietri sul Mare(IT), May 1990.
14. Gori M., Tesi A. On the problem of local minima in backpropagation. IEEE Trans. Pattern Anal. Mach. Intell. 14(1):1992;76-86.
15. Hagan M.T., Menhaj M.B. Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Networks. 5(6):1994;989-993.
16. Horst R., Pardalos P.M. Handbooks of Global Optimization. 1995;Kluwer Academic Publishers, Dordrecht, The Netherlands.
17. C.R. Houch, J.A. Joines, M.G. Kay, A genetic algorithm for function optimization: a matlab implementation, ACM Trans. Math. Software, submitted for Publication.
18. Hwang J., Lay S., Maechler M., Martin R.D., Schimert J. Regression modeling in backpropagation and projection pursuit learning. IEEE Trans. Neural Networks. 5:1994;1-24.
19. Jacobs R.A. Increases rates of convergence through learning rate adaptation. Neural Networks. 1:1988;295-307.
20. K.J. Lang, M.J. Witbrock, Learning to tell two spiral apart, Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.
21. J.R. McDonnell, D. Waagen, Determining neural networks connectivity using evolutionary programming, Twenty-Sixth Asilormar Conference on Signals, Systems, and Computers, Monterey, CA, 1992.
22. Rao S.S. Optimization Theory and Applications. Second ed. 1984;Wiley, New York.
23. S. Renals, R. Rohwer, Phoneme classification experiments using radial basis functions, Proceedings of IEEE International Joint Conference on Neural Networks, Washington, DC, June 1989, pp. I:461-467.
24. Shang Y., Wah B.W. Global optimization for neural network training. IEEE Comput. 29(3):1996;45-54.
25. T. Tolleneace, Super SAB: fast adaptive back propagation with good scaling properties, Neural Networks 3, 561-573.
26. Torn A., Zillinskas A. Global Optimization. 1987;Springler, Berlin.
27. M. Towsey, D. Alpsan, L. Sztriha, Training a neural network with conjugate gradient methods, Proceeding of ICNN'95, Perth, Western Australia, (1995), pp. 373-378.
28. R.J. Williams, Training recurrent networks using the extended kalman filter, International Joint Conference on Neural Networks, Baltimore, 1992, vol. IV, pp. 241-246.
29. Yam Y.F., Chow T.W.S. Extended backpropagation algorithm. Electron. Lett. 29(19):1993;1701-1702.
30. Yam Y.F., Chow T.W.S. Accelerated training algorithm for feedforward neural networks based on linear least squares problems. Neural Process. Lett. 2(4):1995;20-25.
31. Yam Y.F., S Chow T.W. Extended least squares based algorithm for training feedforward networks. IEEE Trans. Neural Networks. 8(3):1997;806-810.
32. Zheng Q., Zhuang D. Integral global optimization of constrained problems in functional spaces with discontinuous penalty functions. Floudas C.A., Pardalos P.M. Recent Advances in Global Optimization. 1992;298-320 Princeton University press, Princeton, NJ.