On the regularization of forgetting recursive least square

IEEE Transactions on Neural Networks

Volumn 10, Issue 6, 1999, Pages 1482-1486

  Leung, Chi Sing ^a   Young, Gilbert H ^b   Sum, John ^c   Kan, Wing Kay ^d  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^b HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

^c HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

^d CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL COMPLEXITY; ERROR ANALYSIS; LEARNING SYSTEMS; LEAST SQUARES APPROXIMATIONS; MATHEMATICAL MODELS;

FORGETTING RECURSIVE LEAST SQUARE (FRLS) TRAINING TECHNIQUES;

FEEDFORWARD NEURAL NETWORKS;

EID: 0033347007     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.809093     Document Type: Article

References (28)

1. A. Barron, "Prediction squared error: A criterion for automatic model selection," Self-Organizing Methods on Modeling, S. Farlow, Ed. New York: Marcel Dekker, 1984.
2. C. M. Bishop, "Improving the generalization properties of radial basis function neural networks," Neural Comput., vol. 3, no. 4, pp. 579-588, 1991.
3. _, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol. 7, no. 1, pp. 108-116, 1995.
4. S. Chen, C. Cowan, S. A. Billings, and P. M. Grant, "Parallel recursive prediction error algorithm for training layered neural networks," Int. J. Contr., vol. 51, no. 6, pp. 1215-1228, 1990.
5. S. Chen, S. A. Billings, C. F. N. Cowan, and P. M. Grant, "Practical identification of NARMAX models using radial basis functions," Int. J. Contr., vol. 52, pp. 1327-1350, 1990.
6. S. Chen, S. A. Billings, and P. M. Grant, "Recursive hybrid algorithm for nonlinear system identification using radial basis function networks," Int. J. Contr., vol. 55, no. 5, pp. 1051-1070, 1992.
7. S. Chen, B. Mulgrew, and P. M. Grant, "A clustering technique for digital communication channel equalization using radial basis function networks," IEEE Trans. Neural Networks, vol. 4, pp. 570-579, 1993.
8. S. Chen and J. Wigger, "Fast orthogonal least squares algorithm for efficient subset model selection," IEEE Trans. Signal Processing, vol. 43, pp. 1713-1715, 1995.
9. D. Gorinevsky, "On the persistency of excitation in radial basis function network identification of nonlinear systems," IEEE Trans. Neural Networks, vol. 6, pp. 1237-1244, 1995.
10. S. Haykin, Neural networks: A Comprehensive Foundation. New York: Macmillan, 1994.
11. T. A. Johansen, "On Tikhonov regularization, bias and variance in nonlinear system identification," Automatica, vol. 33, no. 3, pp. 441-446, 1997.
12. R. Johansson, System Modeling and Identification. Englewood Cliffs, NJ: Prentice-Hall, 1993.
13. S. Kollias and D. Anastassiou, "An adaptive least squares algorithm for the efficient training of artificial neural networks," IEEE Trans. Circuits Syst., vol. 36, pp. 1092-1101, 1989.
14. J. Larsen and L. K. Hansen, "Generalization performance of regularized neural network models," in Proc. IEEE Wkshp. Neural Networks Signal Processing IV, 1994, pp. 42-51.
15. C. S. Leung, K. W. Wong, J. Sum, and L. W. Chan, "On-line training and pruning for RLS algorithms," Electron. Lett., vol. 7, pp. 2152-2153, 1996.
16. C. S. Leung, P. F. Sum, A. C. Tsoi, and L. W. Chan, "Several aspects of pruning methods in recursive least square algorithms for neural networks," in Proc. TANC-97, 1997, pp. 71-80.
17. S. J. Chang, K. W. Wong, and C. S. Leung, "Periodic activation function for fast on-line EKF training and pruning," Electron. Lett., vol. 34, pp. 2255-2256, 1998.
18. J. Sum, C. S. Leung, L. W. Chan, W. K. Kan, and G. Young, "An adaptive Bayesian pruning for neural network in nonstationary environment," Neural Comput., accepted for publication.
19. L. Ljung, J. Sjöberg, and T. McKelvey, "On the use of regularization in system identification," Dept. Elect. Eng., Linkoping University, Sweden, Tech. Rep., 1992.
20. J. Moody, "Note on generalization, regularization, and architecture selection in nonlinear learning systems," in 1st IEEE-SP Wkshp. Neural Networks Signal Processing, 1991.
21. _, "The effective number of parameters: An analysis of generalization regularization in nonlinear learning systems," Advances in Neural Information Processing Systems 4, pp. 847-854, 1992.
22. _, "Prediction risk and architecture selection for neural networks," in From Statistics to Neural Networks: Theory and Pattern Recognition Application, V. Cherkassky et al., Eds. Berlin, Germany: Springer-Verlag, 1994.
23. N. Murata, S. Yoshizawa, and S. Amari, "Network information criterion - Determining the number of hidden units for an artificial neural network model," IEEE Trans. Neural Networks, vol. 5, pp. 865-872, 1994.
24. M. W. Pedersen, L. K. Hansen, and J. Larsen, "Pruning with generalization-based weight saliencies: γOBD, γOBS," Advances in Information Processing Systems 8, pp. 521-528, 1996.
25. S. Shah, F. Palmieri, and M. Datum, "Optimal filtering algorithm for fast learning in feedforward neural networks," Neural Networks, vol. 5, pp. 779-787, 1992.
26. J. Sjöberg and L. Ljung, "Overtraining, regularization and searching for a minimum, with application to neural networks," Int. J. Contr., vol. 62, pp. 1391-1407, 1995.
27. A. N. Tikhonov, "Incorrect problems of linear algebra and a stable method for their solution," Doklady, vol. 163, no. 3, pp. 988-991, 1965.
28. L. Wu and J. Moody, "A smoothing regularizer for feedforward and recurrent neural networks," Neural Comput., vol. 8, pp. 461-489, 1996.