Convergence and objective functions of some fault/noise-injection-based online learning algorithms for RBF networks

IEEE Transactions on Neural Networks

Volumn 21, Issue 6, 2010, Pages 938-947

  Ho, Kevin I J ^a   Leung, Chi Sing ^b   Sum, John ^c  

^a PROVIDENCE UNIVERSITY   (Taiwan)

^b CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^c NATIONAL CHUNG HSING UNIVERSITY   (Taiwan)

Author keywords

Convergence; Fault tolerance; Gladyshev theorem; Objective functions; RBF Networks

Indexed keywords

ALMOST SURE; FAULT-TOLERANT; GLADYSHEV THEOREM; INPUT NOISE; MEAN SQUARE; OBJECTIVE FUNCTIONS; ON-LINE ALGORITHMS; ONLINE LEARNING ALGORITHMS; RBF NETWORK; REGULARIZER; TRAINING ERRORS; WEIGHT DECAY;

ACOUSTIC NOISE; CONVERGENCE OF NUMERICAL METHODS; E-LEARNING; FAULT TOLERANCE; NEURAL NETWORKS; QUALITY ASSURANCE; RADIAL BASIS FUNCTION NETWORKS;

LEARNING ALGORITHMS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; HUMAN; LEARNING; ONLINE SYSTEM; SIGNAL PROCESSING;

ALGORITHMS; HUMANS; LEARNING; NEURAL NETWORKS (COMPUTER); ONLINE SYSTEMS; SIGNAL PROCESSING, COMPUTER-ASSISTED;

EID: 77953119547     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2010.2046179     Document Type: Article

References (49)

1. G. An, "The effects of adding noise during backpropagation training on a generalization performance," Neural Comput., vol.8, no.3, pp. 643-674, Apr. 1996.
2. J. L. Bernier, A. Diaz, F. Fernandez, A. Canas, J. Gonzalez, P. Martin-Smith, and J. Ortega, "Assessing the noise immunity and generalization of radial basis function networks," Neural Process. Lett., vol.18, no.1, pp. 35-48, Aug. 2003.
3. J. L. Bernier, J. Ortega, I. Rojas, and A. Prieto, "Improving the tolerance of multilayer perceptrons by minimizing the statistical sensitivity to weight deviations," Neurocomputing, vol.31, pp. 87-103, Jan. 2000.
4. J. L. Bernier, J. Ortega, I. Rojas, E. Ros, and A. Prieto, "Obtaining fault tolerant multilayer perceptrons using an explicit regularization," Neural Process. Lett., vol.12, no.2, pp. 107-113, Oct. 2000.
5. J. L. Bernier, J. Ortega, E. Ros, I. Rojas, and A. Prieto, "A quantitative study of fault tolerance, noise immunity, and generalization ability of MLPs," Neural Comput., vol.12, no.12, pp. 2941-2964, Dec. 2000.
6. C. M. Bishop, "Training with noise is equivalent to Tikhonov regularization," Neural Comput., vol.7, no.1, pp. 108-116, Jan. 1995.
7. G. Bolt, "Fault tolerant multi-layer perceptron networks," Ph.D. dissertation, Dept. Comput. Sci., Univ. York, York, U.K., Jul. 1992.
8. L. Bottou, "Stochastic gradient learning in neural networks," in Proc. Neuro-Nîmes, Nimes, France, Nov. 1991, pp. 687-706 [Online]. Available: http://leon.bottou.org/papers/bottou-91c
9. L. Bottou, "Online algorithms and stochastic approximations," in Online Learning and Neural Networks, D. Saad, Ed. Cambridge, U.K.: Cambridge Univ. Press, 1998, pp. 9-24 [Online].Available: http://leon. bottou.org/papers/bottou-98x
10. S. Cavalieri and O. Mirabella, "A novel learning algorithm which improves the partial fault tolerance of multilayer neural networks," Neural Netw., vol.12, no.1, pp. 91-106, Jan. 1999.
11. P. Chandra and Y. Singh, "Fault tolerance of feedforward artificial neural networks-A framework of study," in Proc. Int. Joint Conf. Neural Netw., Portland, OR, Jun. 2003, vol.1, pp. 489-494.
12. C. T. Chiu, K. Mehrotra, C. K. Mohan, and S. Ranka, "Modifying training algorithms for improved fault tolerance," in Proc. Int. Conf. Neural Netw., Orlando, FL, Jun. 1994, vol.4, pp. 333-338.
13. D. Deodhare, M. Vidyasagar, and S. Sathiya Keerthi, "Synthesis of fault-tolerant feedforward neural networks using minimax optimization," IEEE Trans. Neural Netw., vol.9, no.5, pp. 891-900, Sep. 1998.
14. P. Edwards and A. Murray, "Fault tolerant via weight noise in analog VLSI implementations of MLP's-A case study with EPSILON," IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process., vol.45, no.9, pp. 1255-1262, Sep. 1998.
15. E. Gladyshev, "On stochastic approximation," Theory Probab. Appl., vol.10, no.2, pp. 275-278, Jan. 1965.
16. Y. Grandvalet, S. Canu, and S. Boucheron, "Noise injection: Theoretical prospects," Neural Comput., vol.9, no.5, pp. 1093-1108, July 1997.
17. N. Hammadi and H. Ito, "A learning algorithm for fault tolerant feedforward neural networks," IEICE Trans. Inf. Syst., vol.80, no.1, pp. 21-27, Jan. 1996.
18. B. Hassibi and D. G. Stork, "Second order derivatives for network pruning: Optimal brain surgeon," in Advances in Neural Information Processing Systems 5. San Francisco, CA: Morgan Kaufmann, 1993, pp. 164-171.
19. S. Himavathi, D. Anitha, and A. Muthuramalingam, "Feedforward neural network implementation in FPGA using layer multiplexing for effective resource utilization," IEEE Trans. Neural Netw., vol.18, no.3, pp. 880-888, May 2007.
20. Y. Jiang, R. Zur, L. Pesce, and K. Drukker, "A study of the effect of noise injection on the training of artificial neural networks," in Proc. Int. Joint Conf. Neural Netw., Jun. 2009, pp. 2784-2788.
21. K. Jim, C. Lee Giles, and B. Horne, "An analysis of noise in recurrent neural networks: Convergence and generalization," IEEE Trans. Neural Netw., vol.7, no.6, pp. 1424-1439, Nov. 1996.
22. N. Kamiura, T. Isokawa, Y. Hara, N. Matsui, and K. Yamato, "On a weight limit approach for enhancing fault tolerance of feedforward neural networks," IEICE Trans. Inf. Syst., vol.83, no.11, pp. 1931-1939, Nov. 2000.
23. T. Lai, "Stochastic approximation," Ann. Stat., vol.31, no.2, pp. 391-406, Apr. 2003.
24. Y. LeCun, J. S. Denker, and S. A. Solla, "Optimal brain damage," in Advances in Neural Information Processing Systems 2. San Mateo, CA: Morgan Kaufmann, 1990, pp. 598-605.
25. C. S. Leung and J. Sum, "A fault-tolerant regularizer for RBF networks," IEEE Trans. Neural Netw., vol.19, no.3, pp. 493-507, May 2008.
26. C. S. Leung, K. W.Wong, P. F. Sum, and L. W. Chan, "On-line training and pruning for RLS algorithms," Electron. Lett., vol.32, no.23, pp. 2152-2153, Nov. 1996.
27. C. S. Leung, K. W. Wong, P. F. Sum, and L. W. Chan, "A pruning method for the recursive least squared algorithm," Neural Netw., vol.14, no.2, pp. 147-174, Mar. 2001.
28. C. S. Leung, G. H. Young, J. Sum, and W. K. Kan, "On the regularization of forgetting recursive least square," IEEE Trans. Neural Netw., vol.10, no.6, pp. 1482-1486, Nov. 1999.
29. A. C. Mallofre and X. P. Llanas, "Fault tolerance parameter model of radial basis function networks," in Proc. Int. Joint Conf. Neural Netw., Jun. 1996, vol.2, pp. 1384-1389.
30. K. Matsuoka, "Noise injection into inputs in back-propagation learning," IEEE Trans. Syst. Man Cybern., vol.22, no.3, pp. 436-440, May 1992.
31. J. E. Moody, "Note on generalization, regularization, and architecture selection in nonlinear learning systems," in Proc. 1st IEEE-SP Workshop Neural Netw. Signal Process., Sep. 1991, DOI: 10.1109/NNSP. 1991.239541 .
32. N. Murata, S. Yoshizawa, and S. Amari, "Network information criterion-determining the number of hidden units for an artificial neural network model," IEEE Trans. Neural Netw., vol.5, no.6, pp. 865-872, Nov. 1994.
33. A. F. Murray and P. J. Edwards, "Synaptic weight noise during multilayer perceptron training: Fault tolerance and training improvements," IEEE Trans. Neural Netw., vol.4, no.4, pp. 722-725, Jul. 1993.
34. A. F. Murray and P. J. Edwards, "Enhanced MLP performance and fault tolerance from synaptic weight noise during training," IEEE Trans. Neural Netw., vol.5, no.5, pp. 792-802, Sep. 1994.
35. C. Neti, M. H. Schneider, and E. D. Young, "Maximally fault tolerance neural networks," IEEE Trans. Neural Netw., vol.3, no.1, pp. 14-23, Jan. 1992.
36. M. W. Pedersen, L. Hansen, and J. Larsen, "Pruning with generalization based weight saliencies: γ-OBD, γ-OBS," in Advances in Neural Information Processing Systems 8. Cambridge, MA: MIT Press, 1995, pp. 521-527.
37. D. S. Phatak, "Relationship between fault tolerance, generalization and the Vapnik-Cervonenkis (vc) dimension of feedforward ANNs," in Proc. Int. Joint Conf. Neural Netw., Jul. 1999, vol.1, pp. 705-709.
38. D. S. Phatak and I. Koren, "Complete and partial fault tolerance of feedforward neural nets," IEEE Trans. Neural Netw., vol.6, no.2, pp. 446-456, Mar. 1995.
39. R. Reed, "Pruning algorithms a survey," IEEE Trans. Neural Netw., vol.4, no.5, pp. 740-747, Sep. 1993.
40. R. Reed, R. J. Marks II, and S. Oh, "Similarities of error regularization, sigmoid gain scaling, target smoothing, and training with jitter," IEEE Trans. Neural Netw., vol.10, no.3, pp. 529-538, May 1995.
41. A. Savich, M. Moussa, and S. Areibi, "The impact of arithmetic representation on implementing MLP-BP on FPGAs: A study," IEEE Trans. Neural Netw., vol.18, no.1, pp. 240-252, Jan. 2007.
42. C. Sequin and R. Clay, "Fault tolerance in artificial neural networks," in Proc. Int. Joint Conf. Neural Netw., Jan. 1990, pp. 703-708.
43. M. Sugiyama and H. Ogawa, "Optimal design of regularization term and regularization parameter by subspace information criterion," Neural Netw., vol.15, no.3, pp. 349-361, Apr. 2002.
44. J. Sum and K. Ho, "SNIWD: Simultaneous weight noise injection with weight decay for MLP training," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, Dec. 2009, vol.5863, pp. 494-501.
45. J. Sum, C. S. Leung, and K. I.-J. Ho, "On objective function, regularizer, and prediction error of a learning algorithm for dealing with multiplicative weight noise," IEEE Trans. Neural Netw., vol.20, no.1, pp. 124-138, Jan. 2009.
46. I. Takanami, "A fault-value injection approach for multiple-weightfault tolerance of MNNs," in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural Netw., Washington, DC, Jul. 2000, vol.3, p. 3515.
47. E. Tchernev, R. Mulvaney, and D. Phatak, "Investigating the fault tolerance of neural networks," Neural Comput., vol.17, no.7, pp. 1646-1664, Jul. 2005.
48. R. Velazco, A. Assoum, N. Radi, R. Ecoffet, and X. Botey, "SEU fault tolerance in artificial neural networks," IEEE Trans. Nuclear Sci., vol.42, no.6, pt. 1, pp. 1856-1862, Dec. 1995.
49. J. Zhu and P. Sutton, "FPGA implementation of neural networks-A survey of a decade of progress," in Proc. 13th Int. Conf. Field Programmable Logic Appl., Sep. 2003, pp. 1062-1066.