Magnified gradient function with deterministic weight modification in adaptive learning

IEEE Transactions on Neural Networks

Volumn 15, Issue 6, 2004, Pages 1411-1423

  Ng, Sin Chun ^a   Cheung, Chi Chung ^b   Leung, Shu Hung ^c  

^a OPEN UNIVERSITY OF HONG KONG   (Hong Kong)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

^c CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Backpropagation (BP); Neural networks

Indexed keywords

ADAPTIVE ALGORITHMS; BACKPROPAGATION; COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; GLOBAL OPTIMIZATION; LEARNING ALGORITHMS;

BACKPROPAGATION LEARNING ALGORITHM; DETERMINISTIC WEIGHT MODIFICATION; GLOBAL CONVERGENCE CAPABILITY; MAGNIFIED GRADIENT FUNCTION;

NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; DECISION SUPPORT SYSTEM; EVALUATION; FEEDBACK SYSTEM; MATHEMATICAL COMPUTING; METHODOLOGY; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SIMULATION; DECISION SUPPORT TECHNIQUES; FEEDBACK; LOGISTIC MODELS; NEURAL NETWORKS (COMPUTER); NUMERICAL ANALYSIS, COMPUTER-ASSISTED; PATTERN RECOGNITION, AUTOMATED;

EID: 9244257332     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2004.836237     Document Type: Article

References (25)

1. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Exploration in the Microstructure of Cognition. Cambridge, MA: MIT Press, 1986, vol. 1.
2. E. K. Blum and L. K. Li, "Approximation theory and feedforward networks," Neural Netw., vol. 4, pp. 511-515, 1991.
3. M. Gori and A. Tesi, "On the problem of local minima in backpropagation," IEEE Trans. Pattern Anal. Machine Intell., vol. 14, pp. 76-86, Jan. 1992.
4. Y. Lee, S. H. Oh, and M. W. Kim, "An analysis of premature saturation in back propagation learning," Neural Netw., vol. 6, pp. 719-728, 1993.
5. F. Stager and M. Agarwal, "Three methods to speed up the training of feedforward and feedback perceptrons," Neural Netw., vol. 10, no. 8, pp. 1435-1443, 1997.
6. A. Van Ooyen and B. Nienhuis, "Improving the convergence of the backpropagation algorithm," Neural Netw., vol. 5, pp. 465-471, 1992.
7. J. E. Vitela and J. Reifman, "Premature saturation in backpropagation networks: Mechanism and necessary conditions," Neural Netw., vol. 10, no. 4, pp. 721-735, 1997.
8. M. Ahmad and F. M. Salam, "Supervised learning using the cauchy energy function," in Proc. 2nd Int. Conf. Fuzzy Logic Neural Networks, Iizuka, Japan, July 17-22, 1992, pp. 721-724.
9. R. A. Jacobs, "Increased rates of convergence through learning rate adaptation," Neural Netw., vol. 1, pp. 295-307, 1988.
10. C. C. Yu and B. D. Liu, "A backpropagation algorithm with adaptive learning rate and momentum coefficient," in Proc. Int. Joint Conf. Neural Neyworks (IJCNN'02), vol. 2, 2002, pp. 1218-1223.
11. X. H. Yu, G. A. Chen, and S. X. Cheng, "Acceleration of backpropagation learning using optimized learning rate and momentum," Electron. Lett., vol. 29, no. 14, pp. 1288-1289, July 8th, 1993.
12. S. E. Fahlman, "Fast learning variations on backpropagation: An empirical study," in Proc. Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, Eds., San Mateo, CA, 1989, pp. 38-51. Pittsburgh, 1988.
13. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. Int. Conf. Neural Networks, vol. 1, 1993, pp. 586-591.
14. N. K. Treadgold and T. D. Gedeon, "Simulated annealing and weight decay in adaptive learning: The SARPROP algorithm," IEEE Trans. Neural Networks, vol. 9, pp. 662-668, July 1998.
15. M. T. Hagan and M. B. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Networks, vol. 5, pp. 989-993, Nov. 1994.
16. D. J. Montana and L. Davis, "Training feedforward neural networks using genetic algorithms," in Proc. Int. Joint Conf. Artificial Intelligence, Detroit, MI, 1989, pp. 762-767.
17. S. Kirkpatrick, "Optimization by simulated annealing: Quantitative studies," J. Statist. Phys., vol. 34, pp. 975-986, 1984.
18. S. Kirkpatrick, C. D. Gilatt, and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, pp. 671-680, 1983.
19. Y. Bard, Nonlinear Parameter Estimation. New York: Academic, 1974.
20. S. Haykin, "Single-layer perceptrons," in Neural Networks: A Comprehensive Foundation, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall, 1999, ch. 3, pp. 117-155.
21. M. N. H. Siddique and M. O. Tokhi, "Training neural networks: Backpropagation vs. genetic algorithms," in Proc. Int. Joint Conf. Neural Networks (IJCNN'01), vol. 4, 2001, pp. 2673-2678.
22. A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and Applications. Melbourne, FL: Krieger, 1980.
23. B. Verma, "Fast training of multilayer perceptions," IEEE Trans. Neural Networks, vol. 8, pp. 1314-1320, Nov. 1997.
24. P. M. Murphy and D. W. Aha. (1994) UCI Repository of Machine Learning Database. Dept. Inform. Comput. Sci., Univ. California, Irvine, CA. [Online]. Available: http://www.ics.uci.edu/~mlearn/ML-Repository.html
25. C. M. Kuan and K. Hornik, "Convergence of learning algorithms with constant learning rates," IEEE Trans. Neural Networks, vol. 2, pp. 484-489, Sept. 1991.