When does online BP training converge?

IEEE Transactions on Neural Networks

Volumn 20, Issue 10, 2009, Pages 1529-1539

  Xu, Zong Ben ^a   Zhang, Rui ^a,b   Jing, Wen Feng ^a  

^a XI'AN JIAOTONG UNIVERSITY   (China)

^b NORTHWEST UNIVERSITY   (China)

Author keywords

Backpropagation (BP) neural networks; Convergence analysis; Online BP training procedure

Indexed keywords

ANALYSIS RESULTS; BACKPROPAGATION (BP) NEURAL NETWORKS; BP NEURAL NETWORKS; CONVERGENCE ANALYSIS; ERROR FUNCTION; FUNDAMENTAL THEOREMS; NEURAL NETWORK LEARNING; ONLINE BP TRAINING PROCEDURE; SCIENTIFIC RESEARCHES; SIGMOID ACTIVATION FUNCTION; STRONG CONVERGENCE; TRAINING PROCEDURES; WEAK CONVERGENCE;

BACKPROPAGATION;

MULTILAYER NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; THEORETICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER);

EID: 70350336479     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2025946     Document Type: Article

References (26)

1. Y. C. Liang, W. Z. Lin, H. P. Lee, S. P. Lim, K. H. Lee, and H. Sun, "Proper orthogonal decomposition and its application - Part II: Model reduction for MEMS dynamical analysis," J. Sound Vib., vol. 256, pp. 515-532, 2002.
2. C. G. Looney, Pattern Recognition Using Neural Networks. New York: Oxford Univ. Press, 1997.
3. T. L. Fine and S. Mukherjee, "Parameter convergence and learning curves for neural networks," Neural Comput., vol. 11, pp. 747-769, 1999.
4. W. Finnoff, "Diffusion approximations for the constant learning rate BP algorithm and resistance to local minima," Neural Comput., vol. 6, no. 2, pp. 285-295, 1994.
5. Z. Luo, "On the convergence of the LMS algorithm with adaptive learning rate for linear feedforward networks," Neural Comput., vol. 3, no. 2, pp. 226-245, 1991.
6. Z. Luo and P. Tseng, "Analysis of an approximate gradient projection method with application to the backpropagation algorithm," Optim. Methods Softw., vol. 4, no. 2, pp. 85-101, 1994.
7. P. Sollich and D. Barber, "Online learning from finite training sets and robustness to input bias," Neural Comput., vol. 10, no. 8, pp. 2201-2217, 1998.
8. D. P. Bertsekas, Nonlinear Programming. Boston, MA: Athena Scientific, 1995.
9. S. H. Oh, "Improving the error bp algorithm with a modified error function, IEEE Trans. Neural Networks," IEEE Trans. Circuits Syst. vol. 8, no. 3, pp. 799-803, May 1997.
10. C. M. Kuan and K. Hornik, "Convergence of learning algorithms with constant learning rates," IEEE Trans. Neural Netw., vol. 2, no. 5, pp. 484-489, Sep. 1991.
11. A. A. Gaivoronski, "Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part I," Optim. Methods Softw., vol. 4, no. 2, pp. 117-134, 1994.
12. O. L. Manggasarian and M. V. Solodov, "Serial and parallel backpropagation convergence via nonmonotone perturbed minimization," Optim. Methods Softw., vol. 4, pp. 103-116, 1994.
13. Z. X. Li, W. Wu, and W. Q. Chen, "Prediction of stock market by BP neural networks with technical indexes as input," J. Math. Res. Explosion, vol. 23, no. 1, pp. 83-97, 2003.
14. H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications. Berlin, Germany: Springer-Verlag, 1997.
15. H. White, "Some asymptotic results for learning in single hidden-layer feedforward neural network models," J. Amer. Statist. Assoc., vol. 84, no. 408, pp. 1003-1013, 1989.
16. W. Wu and Z. Shao, "convergence of an online gradient methods for continuous perceptrons with linearly separable training patterns," Appl. Math. Lett., vol. 16, no. 2, pp. 999-1002, 2003.
17. W.Wu and Y. S. Xu, "Deterministic convergence of an online gradient method for neural networks," J. Comput. Appl. Math., vol. 144, pp. 335-347, 2002.
18. W. Wu, G. R. Feng, and X. Li, "Training multilayer perceptrons via minimization of sum of ridge functions," Adv. Comput. Math., vol. 17, no. 4, pp. 331-347, 2002.
19. Z. Li, W.Wu, and Y. Tian, "Convergence of an online gradient method for FNN with stochastic inputs," J. Comput. Appl. Math., vol. 163, pp. 165-176, 2004.
20. W.Wu, G. R. Feng, Z. X. Li, and Y. S. Xu, "Deterministic convergence of an online gradient method for BP neural networks," IEEE Trans. Neural Netw., vol. 16, no. 3, pp. 533-540, May 2005.
21. Z. X. Li, W. Wu, G. R. Feng, and H. Lu, "Convergence of an online gradient method for BP neural networks with stochastic inputs," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2005, vol. 3601, pp. 720-729.
22. W.Wu, H. Shao, and D. Qu, "Strong convergence for gradient methods for BP networks training," in Proc. Int. Conf. Neural Netw. Brains, 2005, pp. 332-334.
23. N. Zhang, W. Wu, and G. Zheng, "Convergence of gradient method with momentum for two-layer feedforward neural networks with stochastic inputs," IEEE Trans. Neural Netw., vol. 17, no. 2, pp. 522-525, Mar. 2006.
24. D. P. Bertsekas and J. N. Tsitsiklis, "Gradient convergence in gradient methods with errors," SIAM J. Optim., vol. 10, no. 3, pp. 627-642, 2000.
25. J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations With Several Variables. New York: Academic, 1970.
26. S. Haykin, Neural Networks: A Comprehensive Foundation. New York: Macmillan, 1994.