Generalized RLS approach to the training of neural networks

IEEE Transactions on Neural Networks

Volumn 17, Issue 1, 2006, Pages 19-34

  Xu, Yong ^a,b   Wong, Kwok Wo ^a   Leung, Chi Sing ^a  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^b UNIVERSITY OF BIRMINGHAM   (United Kingdom)

Author keywords

Extended Kalman filtering (EFK); Neural network; Recursive least square (RLS) algorithm; Weight decay

Indexed keywords

ALGORITHMS; COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; KALMAN FILTERING; LEAST SQUARES APPROXIMATIONS; MATHEMATICAL MODELS;

EXTENDED KALMAN FILTERING (EFK); QUADRATIC WEIGHT DECAY; RECURSIVE LEAST SQUARE (RLS) ALGORITHM;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; BREAST TUMOR; CLASSIFICATION; COMPUTER SIMULATION; FEMALE; FORECASTING; HUMAN; REGRESSION ANALYSIS; SOLAR ENERGY;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BREAST NEOPLASMS; COMPUTER SIMULATION; FEMALE; FORECASTING; HUMANS; LEAST-SQUARES ANALYSIS; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); SOLAR ACTIVITY;

EID: 33144485895     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2005.860857     Document Type: Article

References (37)

1. C. M. Bishop, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol. 7, pp. 108-116, 1995.
2. L. Wu and J. Moody, "A smoothing regularizer for feedforward and recurrent neural networks," Neural Comput., vol. 8, pp. 461-489, 1996.
3. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Netw., vol. 4, pp. 740-747, 1993.
4. M. Ishikawa, "Structural learning with forgetting," Neural Netw., vol. 9, pp. 509-521, 1996.
5. D. Mackay, "A practical Bayesian framework for backpropagation networks," Neural Comput., vol. 4, pp. 448-472, 1992.
6. C. S. Leung, K. W. Wong, P. F. Sum, and L. W. Chan, "A pruning method for the recursive least squared algorithm," Neural Netw., vol. 14, pp. 147-174, 2001.
7. S. Wang, D. Schuurmans, F. Peng, and Y. Zhao, "Learning mixture models with the regularized latent maximum entropy principle," IEEE Trans. Neural Netw., vol. 16, pp. 903-916, 2004.
8. L. Xu, "Advances on BYY Harmony Learning: Information theoretic perspective, generalized projection geometry, and independent factor autodetermination," IEEE Trans. Neural Netw., vol. 15, pp. 885-902, 2004.
9. Y. Le Cun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Advances in Neural Information Processing 1, D. S. Touretaky, Ed. San Mateo, CA: Morgan Kaufman, 1989, pp. 396-404.
10. H. Robbins and S. Monro, "A stochastic approximation method," Ann. Math. Stat., vol. 22, pp. 400-407, 1951.
11. S. Singhal and L. Wu, "Training feed-forward networks with the extended Kalman filter," in Advances in Neural Information Processing, D. S. Tourezky, Ed. San Mateo, CA: Morgan Kaufman, 1989, pp. 133-140.
12. S. Shah, F. Palmieri, and M. Datum, "Optimal filtering algorithm for fast learning in feedforward neural networks," Neural Netw., vol. 5, pp. 779-787, 1992.
13. A. S. Y. Wong, K. W. Wong, and C. S. Leung, "Recursive least squares approach to combining principal and minor components analyzes," Electron. Lett., vol. 34, pp. 1074-1076, 1998.
14. J. Choi, M. Bouchard, and T. H. Yeap, "Decision feedback recurrent neural equalization with fast convergence rate," IEEE Trans. Neural Netw., vol. 16, pp. 699-708, 2005.
15. C. S. Leung, A. C. Tsoi, and L. W. Chan, "Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks," IEEE Trans. Neural Netw., vol. 12, pp. 1314-1332, 2001.
16. J. Rissanen, "Stochastic complexity in statistical inquiry," in World Scientif., Singapore, 1989.
17. P. C. Cheeseman, "On finding the most probable model," in Computational Model of Scientific Discovery and Theory Formation, J. Shrager and P. Langley, Eds. San Mateo, CA: Morgan Kaufmann, 1990, pp. 73-96.
18. P. L. Bartlett, "The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network," IEEE Trans. Inf. Theory, vol. 4, pp. 525-536, 1998.
19. D. C. Plaut, S. H. Nowlan, and G. E. Elinton, "Experiments on Learning by Back Propagation," Carnegie-Mellon Univ., Tech. Rep. CMU-CS-86-126, 1986.
20. L. K. Hansen and C. E. Rasmussen, "Pruning from adaptive regularization," Neural Comput., vol. 6, pp. 1223-1232, 1994.
21. J. E. Moody, "Note on generalization, regularization and architecture selection in nonlinear systems," in Proc. IEEE Workshop on Neural Networks for Signal Processing, 1991, pp. 1-10.
22. C. Svarer, L. K. Hansen, and J. Larsen, "On design and evaluation of tapped-delay neural network architectures," in IEEE Int. Conf. Neural Networks, vol. 1, San Francisco, CA, 1993, pp. 46-51.
23. C. T. Leung and T. W. S. Chow, "A novel noise robust fourth-order cumulants cost function," Neurocomput., vol. 16, pp. 139-147, 1997.
24. F. Dan Foresee and M. T. Hagan, "Gauss-Newton approximation to Bayesian learning," in Proc. 1997 Int. Joint Conf. Neural Networks, Houston, TX, 1997, pp. 1930-1935.
25. M. T. Hagan and M. B. Manhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, pp. 989-993, 1994.
26. W. H. Wolberg and O. L. Mangasarian, "Multisurface method of pattern separation for medical diagnosis applied to breast cytology," Proc. National Academy of Sciences, vol. 87, pp. 9193-9196, 1990.
27. R. S. Sexton, R. E. Dorsey, and J. D. Johnson, "Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing," Europ. J. Operat. Res., vol. 114, pp. 589-601, 1999.
28. J. T. Corner, R. D. Martin, and L. E. Atlas, "Recurrent neural networks and robust time series prediction," IEEE Trans. Neural Netw., vol. 5, pp. 240-254, 1994.
29. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing. Cambridge, MA: MIT Press, 1986.
30. E. D. Sontag, "Feedforward nets for interpolation and classification," J. Comput. Syst. Sci., vol. 45, pp. 20-48, 1992.
31. T. Ash, "Dynamic node creation," Connection Sci., vol. 1, pp. 365-375, 1989.
32. K. P. Bennett and O. L. Mangasarian, "Neural network training via linear programming," in Advances in Optimization and Parallel Computing, P. M. Pardalos, Ed. Amsterdam, The Netherlands: Elsevier, 1992, pp. 56-67.
33. O. Fujita, "Optimization of the hidden unit function in feedforward neural networks," Neural Netw., vol. 5, pp. 755-764, 1992.
34. M. F. Moller, "A scaled conjugate gradient algorithm for fast supervised learning," Neural Netw., vol. 6, pp. 525-533, 1993.
35. R. Setiono and L. C. K. Hui, "Use of quasi-Newton method in a feedforward neural network construction algorithm," IEEE Trans. Neural Netw., vol. 6, pp. 273-277, 1995.
36. X. Yao and Y. Liu, "A new evolutionary system for evolving artificial neural networks," IEEE Trans. Neural Netw., vol. 8, pp. 694-713, 1997.
37. P. J. Werbos, The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting. New York,: Wiley, 1994.