On the Kalman filtering method in neural-network training and pruning

IEEE Transactions on Neural Networks

Volumn 10, Issue 1, 1999, Pages 161-166

  Sum, John ^a   Leung, Chi Sing ^b   Young, Gilbert H ^c   Kan, Wing Kay ^d  

^a HONG KONG BAPTIST UNIVERSITY   (Hong Kong)

^b NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^c HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

^d CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Extended Kalman filter; Multilayer perceptron; Pruning training; Weight saliency

Indexed keywords

COMPUTER SIMULATION; CONVERGENCE OF NUMERICAL METHODS; ERROR ANALYSIS; KALMAN FILTERING; MULTILAYER NEURAL NETWORKS; PROBLEM SOLVING; SENSITIVITY ANALYSIS;

EXTENDED KALMAN FILTERING METHOD; WEIGHT SALIENCY;

FEEDFORWARD NEURAL NETWORKS;

EID: 0032777345     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.737502     Document Type: Article

References (25)

1. B. D. O. Anderson and J. B. Moore, Optimal Filtering. Englewood Cliffs, NJ: Prentice-Hall, 1979.
2. S. A. Billings and C. F. Fung, "Recurrent radial basis function networks for adaptive noise cancellation," Neural Networks, vol. 8, no. 2, pp. 273-190, 1995.
3. S. Chen, C. F. N. Cowan, S. A. Billings, and P. M. Grant, "Parallel recursive prediction error algorithm for training layered neural network," Int. J. Contr., vol. 51, no. 6, pp. 1215-1228, 1990.
4. S. Chen, S. A. Billing, and P. M. Grant, "A recursive hybrid algorithm for nonlinear system identification using radial basis function networks," Int. J. Contr., vol. 55, no. 5, pp. 1051-1070, 1992.
5. W. Finnoff, F. Hergert, and H. G. Zimmermann, "Improving model selection by nonconvergent methods," Neural Networks, vol. 6, pp. 771-783, 1993.
6. B. Hassibi and D. G. Stork, "Second-order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems, Hanson et al., Eds., 1993, pp. 164-171.
7. S. Haykin, Neural Networks: A Comprehensive Foundation. New York: Macmillan, 1994.
8. Y. Iiguni, H. Sakai, and H. Tokumaru, "A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter," IEEE Trans. Signal Processing, vol. 40, pp. 959-966, 1992.
9. S. Kollias and D. Anastassiou, "An adaptive least squares algorithm for the efficient training of artificial neural networks," IEEE Trans. Circuits Syst., vol. 36, pp. 1092-1101, 1989.
10. Y. LeCun et al., "Optimal brain damage," Advances Neural Inform. Processing Syst. 2, D. S. Touretsky, Ed., pp. 396-404, 1990.
11. C. S. Leung et al., "On-line training and pruning for RLS algorithms," Electron. Lett., No. 23, pp. 2152-2153, 1996.
12. D. G. Luenberger, Introduction to Linear and Nonlinear Programming. Reading, MA, Addison-Wesley, 1973.
13. J. E. Moody, "Note on generalization, regularization, and architecture selection in nonlinear learning systems," in 1st IEEE-SP Wkshp. Neural Networks for Signal Processing, 1991.
14. O. S. P. Bojarczak and M. Stodolski, "Fast second-order learning algorithm for feedforward multilayer neural networks and its application," Neural Networks, vol. 9, no. 9, pp. 1583-1596, 1996.
15. L. Prechelt, "Comparing adaptive and nonadaptive connection pruning with pure early stopping," Progress in Neural Information Processing, pp. 46-52, 1996.
16. G. V. Puskorius and L. A. Feldkamp, "Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks," IEEE Trans. Neural Networks, vol. 5, pp. 279-297, 1994.
17. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Networks, vol. 4, pp. 740-747, 1993.
18. D. Rumelhart, G. Hinton, and G. Williams, "Learning internal representations by error propagation," Parallel Distributed Processing. Cambridge, MA: MIT Press, vol. 1, 1986.
19. S. Shah, F. Palmeieri, and M. Datum, "Optimal filtering algorithms for fast learning in feedforward neural networks," Neural Networks, vol. 5, pp. 779-787, 1992.
20. S. Singhal and L. Wu, "Training multilayer perceptrons with the extended Kalman algorithm," in Advances in Neural Inform. Processing Syst. I, D. S. Touretzky, Ed., pp. 133-140, 1989.
21. J. Sum, "Extended Kalman filter based pruning algorithms and several aspects of neural network learning," Ph.D. dissertation, Dept. Comput. Sci. Eng., Chinese Univ. Hong Kong, July 1998.
22. E. A. Wan and A. T. Nelson, "Dual Kalman filtering methods for nonlinear prediction, smoothing, and estimation," to appear in NIPS'96, 1996.
23. W. K. T. Fukuda and S. G. Tzafestas, "Learning algorithms of layered neural networks via extended Kalman filters," Int. J. Syst. Sci., vol. 22, no. 4, pp. 753-768, 1991.
24. R. J. Williams, "Training recurrent networks using the extended Kalman filter," in Proc. IJCNN'92, Baltimore, 1992, Vol. IV, pp. 241-246.
25. J. M. Zurada, A. Malinowski, and S. Usui, "Perturbation method for deleting redundant inputs of perceptron networks," Neurocomputing, vol. 14, pp. 177-193, 1997.