Deterministic nonmonotone strategies for effective training of multilayer perceptrons

IEEE Transactions on Neural Networks

Volumn 13, Issue 6, 2002, Pages 1268-1284

  Plagianakos, Vassilis P ^a   Magoulas, George D ^b   Vrahatis, Michael N ^a  

^a UNIVERSITY OF PATRAS   (Greece)

^b BRUNEL UNIVERSITY   (United Kingdom)

Author keywords

Adaptive learning rate algorithms; Backpropagation (BP) algorithm; Multilayer perceptrons (MLPs); Nonmonotone minimization; Unconstrained minimization

Indexed keywords

ADAPTIVE ALGORITHMS; BACKPROPAGATION; ERROR ANALYSIS; HEURISTIC METHODS; LEARNING ALGORITHMS;

NONMONOTONE STRATEGIES;

MULTILAYER NEURAL NETWORKS;

EID: 0036859211     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2002.804225     Document Type: Article

References (74)

1. S. Amari, N. Murata, K.-R. Müller, M. Finke, and H.H. Yang, "Asymptotic statistical theory of overtraining and cross-validation," IEEE Trans. Neural Networks, vol. 8, pp. 985-996, Sept. 1997.
2. L. Armijo, "Minimization of functions having Lipschitz-continuous first partial derivatives," Pacific J. Math., vol. 16, pp. 1-3, 1966.
3. A. Atiya and C. Ji, "How initial conditions affect generalization performance in large networks," IEEE Trans. Neural Networks, vol. 8, pp. 448-451, Mar. 1997.
4. J. Barzilai and J.M. Borwein, "Two point step size gradient methods," IMA J. Numer. Anal., vol. 8, pp. 141-148, 1988.
5. R. Battiti, "Accelerated backpropagation learning: Two optimization methods," Complex Syst., vol. 3, pp. 331-342, 1989.
6. _, "First- and second-order methods for learning: Between steepest descent and Newton's method," Neural Comput., vol. 4, pp. 141-166, 1992.
7. S. Becker and Y. Le Cun, "Improving the convergence of the backpropagation learning with second order methods," in Proc. 1988 Connectionist Models Summer School, D.S. Touretzky, G.E. Hinton, and T.J. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1988, pp. 29-37.
8. C.M. Bishop, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol. 7, pp. 108-116, 1995.
9. P. Brodatz, Textures-A Photographic Album for Artists and Designers. New York: Dover, 1966.
10. A. Cauchy, "Méthode générale pour la résolution des systèmes d'équations simultanées," Comp. Rend. Acad. Sci. Paris, vol. 25, pp. 536-538, 1847.
11. L.W. Chan and F. Fallside, "An adaptive training algorithm for back-propagation networks," Comput. Speech Language, vol. 2, pp. 205-218, 1987.
12. J.E. Dennis and J.J. Moré, "Quasi-Newton methods, motivation and theory," SIAM Rev., vol. 19, pp. 46-89, 1977.
13. J.E. Dennis and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Philadelphia, PA: SIAM, 1996.
14. J.L. Elman, E.A. Bates, M.H. Johnson, A. Karmiloff-Smith, D. Parisi, and K. Plunkett, "The shape of change," in Rethinking Innateness: A Connectionist Perspective on Development. Cambridge, MA: MIT Press, 1997, ch. 6.
15. S.E. Fahlman, "Faster-learning variations on back-propagation: An empirical study," in Proc. 1988 Connectionist Models Summer School, D.S. Touretzky, G.E. Hinton, and T.J. Sejnowski, Eds. San Mateo, CA: Morgan Kaufmann, 1988, pp. 38-51.
16. A.V. Fiacco and G.P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques. Philadelphia, PA: SIAM, 1990.
17. O. Fujita, "Statistical estimation of the number of hidden units for feedforward neural networks," Neural Networks, vol. 11, pp. 851-859, 1998.
18. J.C. Gilbert and J. Nocedal, "Global convergence properties of conjugate gradient methods for optimization," SIAM J. Optimization, vol. 2, pp. 21-42, 1992.
19. P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization. New York: Academic, 1981.
20. A.A. Goldstein, "Canchy's method of minimization," Numer. Math., vol. 4, pp. 146-150, 1962.
21. L. Grippo, F. Lampariello, and S. Lucidi, "A nonmonotone line search technique for Newton's method," SIAM J. Numer. Anal., vol. 23, pp. 707-716, 1986.
22. A. Gupta and S.M. Lam, "Weight decay backpropagation for noisy data," Neural Networks, vol. 11, pp. 1127-1137, 1998.
23. M.T. Hagan, H.B. Demuth, and M. Beale, Neural Network Design. Boston, MA: PWS, 1996.
24. R. Haralick, K. Shanmugan, and I. Dinstein, "Textural features for image classification," IEEE Trans. Syst., Man. Cybern., vol. SMC-3, pp. 610-621, 1973.
25. R.M. Haralick, "Statistical and structural approaches to texture," Proc. IEEE, vol. 67, pp. 786-804, 1979.
26. L. Holmstrom and P. Koistinen, "Using additive noise in backpropagation training," IEEE Trans. Neural Networks, vol. 3, pp. 24-38, Jan. 1992.
27. H.C. Hsin, C.C. Li, M. Sun, and R.J. Sclabassi, "An adaptive training algorithm for back-propagation neural networks," IEEE Trans. Syst., Man, Cybern., vol. 25, pp. 512-514, Apr. 1995.
28. R.A. Jacobs, "Increased rates of convergence through learning rate adaptation," Neural Networks, vol. 1, pp. 295-307, 1988.
29. S.A. Karkanis, D.K. Iakovidis, D.E. Maroulis, G.D. Magoulas, and N.G. Theofanous, "Tumor recognition in endoscopic video images using artificial neural network architectures," in Proc. 26th Euromicro Conf., Maastricht, The Netherlands, September 5-7, 2000, pp. 423-429.
30. S.A. Karkanis, G.D. Magoulas, D.K. Iakovidis, D.A. Karras, and D.E. Maronlis, "Evaluation of textural feature extraction schemes for neural network-based interpretation of regions in medical images," in Proc. IEEE Int. Conf. Image Processing (ICIP-2001), vol. 1, Thessaloniki, Greece, 2001, pp. 281-284.
31. S. Karkanis, G.D. Magoulas, and N. Theofanous, "Image recognition and neuronal networks: Intelligent systems for the improvement of imaging information," Minimally Invasive Therapy Allied Technol., vol. 9, pp. 225-230, 2000.
32. G.N. Karystinos and D.A. Pados, "On overfitting, generalization, and randomly expanded training sets," IEEE Trans. Neural Networks, vol. 11, pp. 1050-1057, Sept. 2000.
33. C.M. Kuan and K. Hornik, "Convergence of learning algorithms with constant learning rates," IEEE Trans. Neural Networks, vol. 2, pp. 484-488, Mar. 1991.
34. A.M. Law and W.D. Kelton, Simulation Modeling and Analysis, 3rd ed. New York: MCGraw-Hill, 2000.
35. S. Lawrence, C.L. Giles, and A.C. Tsoi, "What size neural network gives optimal generalization? Convergence properties of backpropagation," Univ. Maryland Tech. Rep. CS-TR-3617, 1996.
36. Y. Le Cun, P.Y. Simard, and B.A. Pearlmutter, "Automatic learning rate maximization by on-line estimation of the Hessian's eigenvectors," in Advances in Neural Information Processing Systems 5, S.J. Hanson, J.D. Cowan, and C.L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 156-163.
37. Y. Lee, S.H. Oh, and M.W. Kim, "An analysis of premature saturation in backpropagation learning," Neural Networks, vol. 6, pp. 719-728, 1993.
38. R. Liu, G. Dong, and X. Ling, "A convergence analysis for neural networks with constant learning rates and nonstationary inputs," in Proc. 34th Conf. Decision Contr., New Orleans, 1995, pp. 1278-1283.
39. C.G. Looney, Pattern Recognition Using Neural Networks. New York: Oxford University Press, 1997.
40. G.D. Magonlas, S.A. Karkanis, D.A. Karras, and M.N. Vrahatis, "Comparison study of textural descriptors for training neural network classifiers," Int. J. Comput. Res., Oct. 2002, to be published.
41. G.D. Magoulas, V.P. Plagianakos, and M.N. Vrahatis, "Hybrid methods using evolutionary algorithms for on-line training," in Proc. IEEE Int. Joint Conf. Neural Networks, Washington, DC, 2001, pp. 2218-2223.
42. G.D. Magoulas, M.N. Vrahatis, and G.S. Androulakis, "Effective back-propagation with variable stepsize," Neural Networks, vol. 10, pp. 69-82, 1997.
43. _, "Improving the convergence of the back-propagation algorithm using learning rate adaptation methods," Neural Computation, vol. 11, pp. 1769-1796, 1999.
44. G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa, and G.S. Androulakis, "Neural network supervised training based on a dimension reducing method," in Mathematics of Neural Networks: Models, Algorithms and Applications, S.W. Ellacot, J.C. Mason, and I.J. Anderson, Eds: Kluwer, 1997, pp. 245-249.
45. K. Matsuoka, "Noise injection into inputs in backpropagation learning," IEEE Trans. Syst., Man, Cybern., vol. 22, pp. 436-440, Mar. 1992.
46. M.F. Møller, "A scaled conjugate gradient algorithm for fast supervised learning," Neural Networks, vol. 6, pp. 525-533, 1993.
47. P.M. Murphy and D.W. Aha. (1994) UCI repository of machine learning databases. Univ. California, Dept. Inform. Comput. Sci., Irvine, CA. [Online]. Available: http://www.ics.uci.edu/mlearn/MLRepository.html.
48. E. Mizutani and S.E. Dreyfus, "On complexity analysis of supervised MLP-learning for algorithmic comparisons," in Proc. Int. Joint Conf. Neural Networks, Washington, DC, 2001, pp. 347-352.
49. D. Nguyen and B. Widrow, "Improving the learning speed of 2-layer neural network by choosing initial values of the adaptive weights," in IEEE Proc. 1st Int. Joint Conf. Neural Networks, vol. 3, 1990, pp. 21-26.
50. J. Nocedal, "Theory of algorithms for unconstrained optimization," Acta Numerica, vol. 1, pp. 199-242, 1992.
51. P.P. Ohanian and R.C. Dubes, "Performance evaluation for four classes of textural features," Pattern Recognition, vol. 25, pp. 819-833, 1992.
52. J.M. Ortega and W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables. New York: Academic, 1970.
53. D.B. Parker, "Optimal algorithms for adaptive networks: Second order back-propagation, second order direct propagation, and second order Hebbian learning," in Proc. IEEE Int. Conf. Neural Networks, 1987, pp. 593-600.
54. M. Pfister and R. Rojas, "Speeding-up backpropagation-A comparison of orthogonal techniques," in Proc. Joint Conf. Neural Networks, Nagoya, Japan, 1993, pp. 517-523.
55. V.P. Plagianakos, D.G. Sotiropoulos, and M.N. Vrahatis, "Automatic adaptation of learning rate for backpropagation neural networks," in Recent Advances in Circuits and Systems, N.E. Mastorakis, Ed. Singapore: World Scientific, 1998, pp. 337-341.
56. V.P. Plagianakos, M.N. Vrahatis, and G.D. Magoulas, "Nonmonotone methods for backpropagation training with adaptive learning rate," in Proc. IEEE Int. Joint Conf. Neural Networks, vol. 3, Washington, DC, 1999, pp. 1762-1767.
57. E. Polak, Optimization: Algorithms and Consistent Approximations. New York: Springer-Verlag, 1997.
58. L. Prechelt, "Automatic early stopping using cross validation: Quantifying the criteria," Neural Networks, vol. 11, pp. 761-767, 1998.
59. M. Raydan, "The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem," SIAM J. Optimization, vol. 7, pp. 26-33, 1997.
60. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The Rprop algorithm," in Proc. IEEE Int. Conf. Neural Networks, San Francisco, CA, 1993, pp. 586-591.
61. A.K. Rigler, J.M. Irvine, and T.P. Vogl, "Rescaling of variables in backpropagation learning," Neural Networks, vol. 4, pp. 225-229, 1991.
62. D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D.E. Rumelhart and J.L. McClelland, Eds. Cambridge, MA: MIT Press, 1986, vol. 1, pp. 318-362.
63. F. Silva and L. Almeida, "Acceleration techniques for the back-propagation algorithm," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 1990, vol. 412, pp. 110-119.
64. A. Sperduti and A. Starita, "Speed up learning and network optimization with extended back-propagation," Neural Networks, vol. 6, pp. 365-383, 1993.
65. J. Strang and T. Taxt, "Local frequency features for texture classification," Pattern Recognition, vol. 27, pp. 1397-1406, 1994.
66. S.B. Thrun, J. Bala, E. Bloedorn, I. Bratko, B. Cestnik, J. Cheng, K. De Jong, S. Dzeroski, S.E. Fahlman, D. Fisher, R. Hamann, K. Kaufmann, S. Keller, I. Kononenko, J. Kreuziger, R.S. Michalski, T. Mitchell, P. Pachowicz, Y. Reich, H. Vafaie, W. Van de Welde, W. Wenzel, J. Wnek, and J. Zhang, "The MONK's problems: A performance comparison of different learning algorithms," Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep., CMU-CS-91-197, 1991.
67. N.K. Treadgold and T.D. Gedeon, "Exploring constructive cascade networks," IEEE Trans. Neural Networks, vol. 10, pp. 1335-1350, Nov. 1999.
68. P.P. Van der Smagt, "Minimization methods for training feedforward neural networks," Neural Networks, vol. 7, pp. 1-11, 1994.
69. M.N. Vrahatis, G.S. Androulakis, J.N. Lambrinos, and G.D. Magoulas, "A class of gradient unconstrained minimization algorithms with adaptive stepsize," J. Comput. Appl. Math., vol. 114, pp. 367-386, 2000.
70. M.N. Vrahatis, G.D. Magoulas, and V.P. Plagianakos, "Globally convergent modification of the Quickprop method," Neural Processing Lett., vol. 12, pp. 159-169, 2000.
71. T.P. Vogl, J.K. Mangis, J.K. Rigler, W.T. Zink, and D.L. Alkon, "Accelerating the convergence of the back-propagation method," Biol. Cybern., vol. 59, pp. 257-263, 1988.
72. R.L. Watrous, "Learning algorithms for connectionist networks: Applied gradient of nonlinear optimization," in Proc. IEEE Int. Conf. Neural Networks, vol. 2, 1987, pp. 619-627.
73. S.A. Karkanis, D.K. Iakovidis, D.E. Maroulis, G.D. Magoulas, and N.G. Theofanous, "Tumor recognition in endoscopic video images using artificial neural network architectures," in Proc. 26th Euromicro Conf., F. Vajda, Ed. Los Alamitos, CA: IEEE Press, 2000.
74. S.A. Karkanis, G.D. Magoulas, D.K. Iakovidis, D.A. Karras, and D.E. Maroulis, "Evaluation of textural feature extraction schemes for neural network-based interpetation of regions in medical images," in Proc. IEEE Int. Conf. Image Processing (ICIP-2001). Los Alamitos, CA: IEEE Press, 2001.