An online backpropagation algorithm with validation error-based adaptive learning rate

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 4668 LNCS, Issue PART 1, 2007, Pages 249-258

  Duffner, Stefan ^a   Garcia, Christophe ^a  

^a ORANGE LABS   (France)

Author keywords

[No Author keywords available]

Indexed keywords

E-LEARNING; ERRORS; FEEDFORWARD NEURAL NETWORKS; HEURISTIC METHODS; ITERATIVE METHODS; LEARNING SYSTEMS;

ADAPTIVE LEARNING RATES; EARLY STOPPING; GLOBAL LEARNING; LEARNING RATES; SECOND PHASE; TRAINING SETS; VALIDATION ERRORS;

BACKPROPAGATION;

EID: 38149100484     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-540-74690-4_26     Document Type: Conference Paper

References (20)

1. Rumelhart, D.E., McClelland, J.L.: Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Foundations, vol. 1. MIT Press, Cambridge, MA, USA (1986)
2. Chan, LW., Fallside, F.: An adaptive learning algorithm for backpropagation networks. Computer Speech and Language 2, 205-218 (1987)
3. Salomon, R.: Improved convergence rate of back-propagation with dynamic adaptation of the learning rate. In: Schwefel, H.-P., Männer, R. (eds.) Parallel Problem Solving from Nature. LNCS, vol. 496, pp. 269-273. Springer, Heidelberg (1991)
4. Battiti, R.: Accelerated backpropagation learning: Two optimization methods. Complex Systems 3, 331-342 (1989)
5. Vogl, T., Mangis, J., Rigler, J., Zink, W., Alkon, D.: Accelerating the convergence of the back-propagation method. Biological Cybernetics 59, 257-263 (1988)
6. Hsin, H.C., Li, C.C., Sun, M., Sclabassi, R.: An adaptive training algorithm for back-propagation neural networks. In: International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1049-1052 (1992)
7. Palgianakos, V., Vrahatis, M., Magoulas, G.: Nonmonotone methods for backpropagation training with adaptive learning rate. In: International Joint Conference on Neural Networks, vol. 3, pp. 1762-1767 (1999)
8. LeCun, Y., Simard, P., Pearlmutter, B.: Automatic learning rate maximization by on-line estimation of the hessian's eigenvectors. In: Hanson, S., Cowan, J., Giles, L. (eds.) Advances in Neural Information Processing Systems, vol. 5, Morgan Kaufmann Publishers, San Mateo, CA (1993)
9. Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Effective back-propagation with variable stepsize. Neural Networks 10, 69-82 (1997)
10. Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Improving the convergence of the backpropagation algorithm using learning rate adaptation methods. Neural Computation 11(7), 1769-1796 (1999)
11. Jacobs, R.A.: Increased rates of convergence through learning rate adaption. Neural Networks 1, 295-307 (1988)
12. Silva, F.M., Almeida, L.B.: Acceleration techniques for the backpropagation algorithm. In: Proceedings of the EURASIP Workshop 1990 on Neural Networks, London, UK, pp. 110-119. Springer, Heidelberg (1990)
13. Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Proc. of the IEEE Intl. Conf. on Neural Networks, San Francisco, CA, pp. 586-591. IEEE Computer Society Press, Los Alamitos (1993)
14. Luenberger, D.G.: Introduction to linear and nonlinear programming. Addison-Wesley, Reading (1973)
15. Fahlman, S.E.: Faster-learning variations on back-propagation: An empirical study. In: Touretzky, D.S., Hinton, G.E., Sejnowski, T.J. (eds.) Connectionist Models Summer School, pp. 38-51. Morgan Kaufmann Publishers, San Mateo, CA (1988)
16. Leonard, J., Kramer, M.: Improvement of the backpropagation algorithm for training neural networks. Computers & Chemical Engineering 14(3), 337-341 (1990)
17. Schraudolph, N.: Local gain adaptation in stochastic gradient descent. In: ICANN. Ninth International Conference on Artificial Neural Networks, vol. 2, pp. 569-574 (1999)
18. Harmon, M., Baird III, L.: Multi-player residual advantage learning with general function approximation. Technical report, Wright Laboratory, WL/AACF, Wright-Patterson Air Force Base, OH (1996)
19. Almeida, L.B., Langlois, T., Amaral, J.D., Plakhov, A.: Parameter adaptation in stochastic optimization. In: Saad, D. (ed.) On-Line Learning in Neural Networks, Cambridge University Press, Cambridge (1999)
20. Sutton, R.S.: Adapting bias by gradient descent: an incremental version of belta-bar-delta. In: Proceedings of the Tenth International Conference on Machine Learning, Cambridge, MA, pp. 171-176 (1992)