A comparison of nonlinear optimization methods for supervised learning in multilayer feedforward neural networks

European Journal of Operational Research

Volumn 93, Issue 2, 1996, Pages 358-368

  Denton, James W ^a   Hung, Ming S ^b  

^a WEST VIRGINIA UNIVERSITY   (United States)

^b KENT STATE UNIVERSITY   (United States)

Author keywords

Neural network training; Nonlinear programming

Indexed keywords

ALGORITHMS; DATA STORAGE EQUIPMENT; LEARNING SYSTEMS; NONLINEAR PROGRAMMING; OPTIMIZATION; PATTERN RECOGNITION;

CONJUGATE GRADIENT; MULTILAYER FEEDFORWARD NEURAL NETWORKS; PATTERN CLASSIFICATION; SUPERVISED LEARNING;

FEEDFORWARD NEURAL NETWORKS;

EID: 0030572649     PISSN: 03772217     EISSN: None     Source Type: Journal    
DOI: 10.1016/0377-2217(96)00035-5     Document Type: Article

References (35)

1. Bai, B., and Farhat, N.H., "Learning neural networks for extrapolation and radar target identification", Neural Networks 5 (1992) 507-529.
2. Barnard, E., "Optimization for training neural nets", IEEE Transactions on Neural Networks 3 (1992) 232-240.
3. Battiti, R., "First- and second-order methods for learning: between steepest descent and Newton's method", Neural Computation 4 (1992) 141-166.
4. Biegler-Konig, F., and Barmann, F., "A learning algorithm for multilayered neural networks based on linear least squares problems", Neural Networks 6 (1993) 127-131.
5. Broyden, C.G., "Convergence of a class of double rank minimization algorithms", Journal of the Institute of Mathematics and Its Applications 6 (1970) 76-90.
6. Fisher, R.A., "The use of multiple measurements in taxonomic problems", Annals of Eugenics 1 (1936) 179-188.
7. Fishman, M.B., Barr, D.S., and Loick, W.J., "Using neural nets in market analysis", Technical Analysis of Stocks and Commodities, 1991, 18-25.
8. Fletcher, R., Practical Methods of Optimization, Wiley, Chichester, 1987.
9. Fletcher, R., "A new approach to variable metric algorithms", Computer Journal 13 (1970) 317-322.
10. Fletcher, R., and Reeves, C.M., "Function minimization by conjugate gradients", Computer Journal 7 (1964) 149-154.
11. Gill, P.E., Murray, W., and Wright, M.H., Practical Optimization, Academic Press, London, 1981.
12. Goldfarb, D., "Factorized variable metric methods for unconstrained optimization", Mathematics of Computation 30 (1976) 796-811.
13. Goldfarb, D., "A family of variable metric methods derived by variational means", Mathematics of Computation 24 (1970) 23-26.
14. Goldstein, A.A., "On steepest descent", SIAM Journal of Control 3 (1965) 147-151.
15. Gorman, R.P., and Sejnowski, T.J., "Analysis of hidden units in a layered network trained to classify sonar targets". Neural Networks 1 (1988) 75-89.
16. Himmelblau, D.M., "Introducing efficient second order effects into back propagation learning", in: Proceedings of the International Joint Conference on Neural Networks, IEEE, New York, 1990, I-631-634.
17. Hung, M.S., and Denton, J.W., "Training neural networks with the GRG2 nonlinear optimizer", European Journal of Operational Research 69 (1993) 83-91.
18. Kinsella, J.A., "Comparison and evaluation of variants of the conjugate gradient method for efficient learning in feed-forward neural networks with backward error propagation", Network 3 (1991) 27-35.
19. Koster, A., Sondak, N.E., and Bourbia, W., "A business application of artificial neural network systems", The Journal of Computer Information Systems 31 (1991) 3-9.
20. Liang, T., Moskowitz, H., and Yih, Y., "Integrating neural networks and semi-Markov processes for automated knowledge acquisition: an application to real-time scheduling", Decision Sciences 23 (1992) 1297-1314.
21. Mendelsohn, L., and Stein, J., "Fundamental analysis meets the neural network", Futures 20 (1991) 22-24.
22. Moller, M., "A scaled conjugate gradient algorithm for fast supervised learning", Neural Networks 6 (1993) 525-533.
23. Mukhopadhyay, S., Roy, A., Kim, L.S., and Govil, S., "A polynomial time algorithm for generating neural networks for pattern classification: Its stability properties and some test results", Neural Computation 5 (1993) 317-330.
24. Nocedal, J., "Updating quasi-Newton matrices with limited storage", Mathematics of Computation 35 (1980) 773-752.
25. Powell, M.J.D., "Some global convergence properties of a variable metric algorithm for minimization without exact line searches", in: Cottle, R.W., and Lemke, C.E. (eds.), SIAM-AMS Proceedings, vol. XI, SIAM Publications, Philadelphia, 1976.
26. Rumelhart, D.E., Hinton, G.E., and Williams, R.J., "Learning internal representations by error propagation", in: Rumelhart, D.E., and McClelland, J.L., (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations, MIT Press, Cambridge, MA, 1986, 18-362.
27. Säckinger, E., Boser, B.E., Bromley, J., Le Cun, Y., and Jacket, L., "Applications of the ANNA neural network chip to high-speed character recognition", IEEE Transactions on Neural Networks 3 (1992) 498-505.
28. Salchenberger, L.M., Cinar, E.M., and Lash, N.A., "Neural networks: A new tool for predicting thrift failures", Decision Sciences 23 (1992) 899-916.
29. Shanno, D., "Conditioning of quasi-Newton methods for function minimization", Mathematics of Computation 24 (1970) 647-657.
30. Sharda, R., "Neural Networks for the MS/OR analyst: An application bibliography", Interfaces 24 (1994) 116-130.
31. Sperduti, A., and Starita, A., "Speed up learning and network optimization with extended back propagation", Neural Networks 6 (1993) 365-383.
32. Subramanian, V., and Hung, M.S., "A GRG2-based system for training neural networks: Design and computational experience", ORSA Journal on Computing 5 (1993) 386-394.
33. Tam, K.Y., and Kiang, M.Y., "Managerial applications of neural networks: The case of bank failure predictions", Management Science 38 (1992) 926-947.
34. Watrous, R.L., "Learning algorithms for connectionist networks: Applied gradient methods of nonlinear optimization", in: Proceedings of IEEE International Conference on Neural Networks, IEEE, New York, 1987, II-619-II-627.
35. Wolfe, P., "Convergence conditions for ascent methods", SIAM Review 11 (1968) 226-235.