A class of gradient unconstrained minimization algorithms with adaptive stepsize

Journal of Computational and Applied Mathematics

Volumn 114, Issue 2, 2000, Pages 367-386

  Vrahatis, M N ^a   Androulakis, G S ^a   Lambrinos, J N ^a   Magoulas, G D ^b  

^a UNIVERSITY OF PATRAS   (Greece)

^b UNIVERSITY OF ATHENS   (Greece)

Author keywords

49D37; 65C20; 65K05; 65K10; 68T05; 82C32; Armijo's method* Globally convergent method; Artificial neural network; Gradient method* Lipschitz constant; Line search strategies; Steepest descent; Training algorithm; Unconstrained optimization

Indexed keywords

CONVERGENCE OF NUMERICAL METHODS; NEURAL NETWORKS; OPTIMIZATION;

ADAPTIVE STEPSIZE; GRADIENT UNCONSTRAINED MINIMIZATION ALGORITHMS; LIPSCHITZ CONSTANT;

ALGORITHMS;

EID: 0033899834     PISSN: 03770427     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0377-0427(99)00276-9     Document Type: Article

References (49)

1. Androulakis G.S., Vrahatis M.N. OPTAC. a portable software package for analyzing and comparing optimization methods by visualization J. Comput. Appl. Math. 72:1996;41-62.
2. Androulakis G.S., Vrahatis M.N., Grapsa T.N. Studying the performance of optimization methods by visualization. Systems Anal. Model. Simulation. 25:1996;21-42.
3. Armijo L. Minimization of function having Lipschitz continuous first partial derivatives. Pacific J. Math. 16:1966;1-3.
4. O. Axelsson, Iterative Solution Methods, Cambridge University Press, New York, 1996.
5. Blum E.K. Approximation of Boolean functions by sigmoidal networks. part I: XOR and other two-variable functions Neural Comput. 1:1989;532-540.
6. R.P. Brent, Algorithms for Minimization Without Derivatives, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1973.
7. Brewster M.E., Kannan R. Nonlinear successive over-relaxation. Numer. Math. 44:1984;309-315.
8. P. Brodatz, Textures - A Photographic Album for Artists and Designers, Dover, New York, 1966.
9. R.H. Byrd, J. Nocedal, A tool for the analysis of quasi-Newton methods with application to unconstrained minimization, SIAM J. Numer. Anal. (1989) 727-739.
10. Curry H.B. The method of steepest descent for non-linear minimization problems. Quart. Appl. Math. 2:1944;258-261.
11. Cauchy A. Méthode générale pour la résolution des systèmes d'équations simultanées. Comp. Rend. Acad. Sci. Paris. 25:1847;536-538.
12. E.K.P. Chong, S.H. Żak, An Introduction to Optimization, Wiley, New York, 1996.
13. J.E. Dennis Jr., R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Inc., Englewood Cliffs, NJ, 1983.
14. Fletcher R., Reeves C. Function minimization by conjugate gradients. Comput. J. 7:1964;149-154.
15. Gilbert J.C., Nocedal J. Global convergence properties of conjugate gradient methods for optimization. SIAM J. Optim. 2:1992;21-42.
16. Goldstein A.A. Cauchy's method of minimization. Numer. Math. 4:1962;146-150.
17. Goldstein A.A. On steepest descent. SIAM J. Control. 3:1965;147-151.
18. Goldstein A.A., Price J.F. An effective algorithm for minimization. Numer. Math. 10:1967;184-189.
19. Grapsa T.N., Vrahatis M.N. A dimension-reducing method for unconstrained optimization. J. Comput. Appl. Math. 66:1996;239-253.
20. A.V. Fiacco, G.P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques, SIAM, Philadelphia, 1990.
21. Haralick R., Shanmugan K., Dinstein I. Textural features for image classification. IEEE Trans. System, Man Cybernet. 3:1973;610-621.
22. S. Haykin, Neural Networks: A Comprehensive Foundation, Macmillan College Publishing Company, New York, 1994.
23. Heller D. A survey of parallel algorithms in numerical linear algebra. SIAM Rev. 20:1978;740-777.
24. Jacobs R.A. Increased rates of convergence through learning rate adaptation. Neural Networks. 1:1988;295-307.
25. G.D. Magoulas, M.N. Vrahatis, G.S. Androulakis, A new method in neural network supervised training with imprecision, Proceedings of the IEEE Third International Conference on Electronics, Circuits and Systems, 1996 , pp. 287 - 290 .
26. Magoulas G.D., Vrahatis M.N., Androulakis G.S. Effective backpropagation training with variable stepsize. Neural Networks. 10:1997;69-82.
27. G.D. Magoulas, M.N. Vrahatis, G.S. Androulakis, Improving the convergence of the backpropagation algorithm using learning rate adaptation methods, Neural Computation 11 (1999) 1769-1796.
28. Moré B.J., Garbow B.S., Hillstrom K.E. Testing unconstrained optimization. ACM Trans. Math. Software. 7:1981;17-41.
29. Nocedal J. Theory of algorithms for unconstrained optimization. Acta Numerica. 1:1992;199-242.
30. Ohanian P.P., Dubes R.C. Performance evaluation for four classes of textural features. Pattern Recognition. 25:1992;819-833.
31. J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, 1970.
32. Ortega J.M., Voigt R.G. Solution of partial differential equations on vector and parallel computers. SIAM Rev. 27:1985;149-270.
33. E. Polak, Computational Methods in Optimization, Academic Press, New York, 1971.
34. E. Polak, Optimization: Algorithms and Consistent Approximations, Springer, New York, 1997.
35. Powell M.J.D. Direct search algorithms for optimization calculations. Acta Numerica. 7:1998;287-336.
36. S.S. Rao, Optimization, Theory and Applications 2nd Edition, 7th reprint, Wiley Eastern Limited, New Delhi, 1992.
37. D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning internal representations by error propagation, in: D.E. Rumelhart, J.L. McClelland (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1, MIT Press, 1986, pp. 318-362.
38. Sperduti A., Starita A. Speed up learning and network optimization with extended back-propagation. Neural Networks. 6:1993;365-383.
39. G.W. Stewart, Introduction to Matrix Computations, Academic Press, New York, 1973.
40. Strang J., Taxt T. Local frequency features for texture classification. Pattern Recognition. 27:1994;1397-1406.
41. R. Varga, Matrix Iterative Analysis, Prentice-Hall, Inc., Englewood Cliffs, NJ, 1962.
42. Vogl T.P., Mangis J.K., Rigler A.K., Zink W.T., Alkon D.L. Accelerating the convergence of the back-propagation method. Biol. Cybernet. 59:1988;257-263.
43. Voigt R.G. Rates of convergence for a class of iterative procedures. SIAM J. Numer. Anal. 8:1971;127-134.
44. Vrahatis M.N., Androulakis G.S., Manoussakis G.E. A new unconstrained optimization method for imprecise function and gradient values. J. Math. Anal. Appl. 197:1996;586-607.
45. Wessel L.F., Barnard E. Avoiding false local minima by proper initialization of connections. IEEE Trans. Neural Networks. 3:1992;899-905.
46. Wolfe P. Convergence conditions for ascent methods. SIAM Rev. 11:1969;226-235.
47. Wolfe P. Convergence conditions for ascent methods II. some corrections SIAM Rev. 13:1971;185-188.
48. Young D. Iterative methods for solving partial difference equations of elliptic type. Trans. Amer. Math. Soc. 76:1954;92-111.
49. G. Zoutendijk, Nonlinear programming, computational methods, in: J. Abadie (Ed.), Integer and Nonlinear Programming, North-Holland, Amsterdam, 1970, pp. 37-86.