Acceleration of the back propagation through dynamic adaptation of the learning rate

International Journal of Computer Mathematics

Volumn 65, Issue 1-2, 1997, Pages 1-17

  Zainuddin, Z ^a   Evans, D J ^b  

^a SCHOOL OF PHYSICS   (Malaysia)

^b LOUGHBOROUGH UNIVERSITY   (United Kingdom)

Author keywords

Backpropagation; Convergence; Learning algorithm; Steepest descent

Indexed keywords

ADAPTIVE ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; ITERATIVE METHODS; LEARNING ALGORITHMS; NEURAL NETWORKS; OPTIMIZATION;

STEEPEST DESCENT;

BACKPROPAGATION;

EID: 0031346709     PISSN: 00207160     EISSN: None     Source Type: Journal    
DOI: 10.1080/00207167708804597     Document Type: Article

References (16)

1. Battiti, R. (1992). First and Second Order Methods for Learning: Between Steepest Descent and Newton's Method, Neural Computation, 4, 141-166.
2. Becker, S. and Le Cun, Y. (1989). Improving the convergence of back-propagation learning with second order methods. In Touretzky, D., Hinton, G. and Sejnowski, T. (Eds.). Proceedings of the 1988 Connectionist Models Summer School (pp.29-37). San Mateo, CA: Morgan kaufmann.
3. Carter, J. P. (1987). Successfully using Peak Learning Rates of 10 (and greater) in the Back-Propagation Networks with the HEURISTIC Llearning Algorithm, Proceedings of IEEE, Ist Int. Conf. on NN, San Diego, CA, pp. 645-651.
4. Evans, D. J. and Sanossian, H. Y. Y. (1993). A Gradient Range Bases Heuristic Algorithm for Back. Propagation, J. of Microcomputer Application, 16, pp. 179-188.
5. Fahlman, S.E (1989). Faster Learning Varitation on Back Propagation. An Empirical Study, Proceedings of the 1988 Connectionist Models, Kaufman, pp.38-51.
6. Hush, D. R. and Salas, J. M. Improving the Learning Rate of Back-Propagation with the Gradient Reuse Algorithm, IEEE Int. Conf. on neural Networks, 1, pp. 441-447, san Diego, CA.
7. Jacobs, R. A. (1988). Increased Rates of Convergence through Learning Rate Adaptation, Neural Networks, 1(4), 295-308.
8. Kramer, A. H. and Sangiovanni-Vincentelli, A. (1989) Efficient parallel learning algorithms for neural networks, In Touretzky, D. S. (Ed.), Advances in Neural Information systems 1, (pp.40-48). San Mateo, CA: Morgan Kaufmann.
9. Plaut, D. C., Nowlan, S. J. and Hinton, G. E. (1989). Experiments on Learning Back Propagation, Technical Report CMU-CS, 86-126, Carnegie-Mellon University, Pittusburg, PA.
10. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning Internal Representations by Error Propagation., In Rumelhart, D.E., McClelland, J.L., (eds.). Parallel Distributed Processing:Explorations in the Microstructure of Cognition, pp. 318-362. MIT press, Cambridge, MA.
11. Salomon, R. Improved convergence rate of back-propagation with dynamic adaptation of the learning rate.
12. Sanossian, H. Y. Y. (1993). Neural Network Learning Algorithms, PhD Thesis, Loughborough University.
13. Silva, F. M. and Almeida, L. B. (1990). Acceleration Techniques for the Back Propagation Algorithm, Neural Networks EURASIP Workshop, Sesim.
14. Sutton, R. S. (1986). Two Problems with Back Propagation and Other Steepest Descent Learning Procedures for Networks, Proceedings of the 8th ANN Conference of the Cognitive Science Society, pp.823-831.
15. Van Der Smagt, P. P. (1994). Minimisation Methods for Training Feedforward Neural Networks, Neural Networks, 7(1), pp.1-11.
16. Watrous, R. L. (1987). Learning Algorithms for Connectionist Networks:Applied Gradient Methods of Nonlinear Optimization, Proceedings of the IEEE 1st International Conference on Neural Networks, San Diego, CA, pp. 619-627.