Boundedness and convergence of online gadient method with penalty for feedforward neural networks

IEEE Transactions on Neural Networks

Volumn 20, Issue 6, 2009, Pages 1050-1054

  Zhang, Huisheng ^a,b   Wu, Wei ^a   Liu, Fei ^c   Yao, Mingchen ^a  

^a DALIAN UNIVERSITY OF TECHNOLOGY   (China)

^b DALIAN MARITIME UNIVERSITY   (China)

^c UNIVERSITY OF MISSOURI   (United States)

Author keywords

Boundedness; Convergence; Feedforward neural networks; Online gradient method; Penalty

Indexed keywords

BOUNDEDNESS; CONVERGENCE; GENERALIZATION PERFORMANCE; NETWORK TRAINING; NUMERICAL EXAMPLE; ONLINE GRADIENT METHOD; PENALTY;

GRADIENT METHODS;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER SIMULATION; METHODOLOGY; THEORETICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 67649385962     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2020848     Document Type: Article

References (21)

1. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
2. D. P. Bertsekas and J. N. Tsitsiklis, "Gradient convergence in gradient methods with errors," SIAM J. Optim., vol. 3, pp. 627-642, 2000.
3. P. Billingsley, Probability and Measure. New York: Wiley, 1979.
4. T. Evgeniou, M. Pontil, and T. Poggio, "Regularization networks and support vector machines," Adv. Comput. Math., vol. 13, pp. 1-50, 2000.
5. T. L. Fine and S. Mukherjee, "Parameter convergence and learning curves for neural networks," Neural Comput., vol. 11, pp. 747-769, 1999.
6. A. A. Gaivoronski, "Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods (Part I)," Optim. Methods Software, vol. 4, pp. 117-134, 1994.
7. L. Grippo, "Convergent on-line algorithms for supervised learning in neural networks," IEEE Trans. Neural Netw., vol. 11, no. 6, pp. 1284-1299, Nov. 2000.
8. S. J. Hanson and L. Y. Pratt, "Comparing biases for minimal network construction with back-propagation," Adv. Neural Inf. Process., vol. 1, pp. 177-185, 1989.
9. E. D. Karnin, "A simple procedure for pruning back-propagation trained neural networks," IEEE Trans. Neural Netw., vol. 1, no. 2, pp. 239-242, Jun. 1990.
10. L. Ljung, "Analysis of recursive stochastic algorithm," IEEE Trans. Autom. Control, vol. AC-22, no. 4, pp. 551-575, Aug. 1977.
11. O. L. Mangasarian and M. V. Solodov, "Serial and parallel backpropagation convergence via nonmonotone perturbed minimization," Optim. Methods Software, vol. 4, pp. 117-134, 1994.
12. S. C. Ng, C. C. Cheung, and S. H. Leung, "Magnified gradient function with deterministic weight modification in adaptive learning," IEEE Trans. Neural Netw., vol. 15, no. 6, pp. 1411-1423, Nov. 2004.
13. S. J. Nowlan and G. E. Hinton, "Simplifying neural networks by soft weight sharing," Neural Comput., vol. 4, pp. 173-193, 1992.
14. D. C. Plaut, S. J. Nowlan, and G. E. Hinton, "Experiments on learning by back-propagation," Carnegie-Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-86-126, 1986.
15. R. Reed, "Pruning algorithms: A survey," IEEE Trans. Neural Netw. vol. 4, no. 5, pp. 740-747, Sep. 1993.
16. K. Saito and R. Nakano, "Second-order learning algorithm with squared penalty term," Neural Comput., vol. 12, pp. 709-729, 2000.
17. H. Shao, W. Wu, and L. Liu, "Convergence and monotonicity of an online gradient method with penalty for neural networks," WSEAS Trans. Math., vol. 6, pp. 469-476, 2007.
18. V. Tadic and S. Stankovic, "Learning in neural networks by normalized stochastic gradient algorithm: Local convergence," in Proc. 5th Seminar Neural Netw. Appl. Electr. Eng., Yugoslavia, Sep. 2000, pp. 11-17.
19. H. White, "Some asymptotic results for learning in single hidden-layer feedforward network models," J. Amer. Statist. Assoc., vol. 84, pp. 1003-1013, 1989.
20. W. Wu, G. Feng, and X. Li, "Training multilayer perceptrons via minimization of sum of ridge functions," Adv. Comput. Math., vol. 17, pp. 331-347, 2002.
21. W. Wu, G. Feng, Z. Li, and Y. Xu, "Deterministic convergence of an online gradient method for BP neural networks," IEEE Trans. Neural Netw., vol. 16, no. 3, pp. 533-540, May 2005.