Real-time learning capability of neural networks

IEEE Transactions on Neural Networks

Volumn 17, Issue 4, 2006, Pages 863-878

  Huang, Guang Bin ^a   Zhu, Qin Yu ^a   Siew, Chee Kheong ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

Author keywords

NN; Backpropagation (BP); Extreme learning machine; Feedforward networks; Generalization performance; Real time learning; Real time prediction

Indexed keywords

ARTIFICIAL INTELLIGENCE; BACKPROPAGATION; FEEDFORWARD NEURAL NETWORKS; GRADIENT METHODS; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATHEMATICAL MODELS; REGRESSION ANALYSIS;

EXTREME LEARNING MACHINE; GENERALIZATION PERFORMANCE; REAL-TIME LEARNING; REAL-TIME PREDICTION;

NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SYSTEM; LEARNING;

COMPUTER SYSTEMS; LEARNING; NEURAL NETWORKS (COMPUTER);

EID: 33746916409     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2006.875974     Document Type: Article

References (32)

1. G.-B. Huang, "Learning capability and storage capacity of two-hidden-layer feedforward networks," IEEE Trans. Neural Netw., vol. 14, no. 2, pp. 274-281, Mar. 2003.
2. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Netw., vol. 2, pp. 359-366, 1989.
3. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Control, Signals Syst., vol. 2, no. 4, pp. 303-314, 1989.
4. K. Funahashi, "On the approximate realization of continuous mappings by neural networks," Neural Netw., vol. 2, pp. 183-192, 1989.
5. A. Gallant and H. White, "There exists a neural network that does not make avoidable mistakes," in Artificial Neural Networks: Approximation and Learning Theory, H. White, Ed. Oxford, U.K.: Blackwell, 1992, pp. 5-11.
6. M. Stinchcombe and H. White, "Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions," in Artificial Neural Networks: Approximation and Learning Theory, H. White, Ed. Oxford, U.K.: Blackwell, 1992, pp. 29-40.
7. d by linear combinations of shifted rotations of a sigmoid function with and without scaling," Neural Netw., vol. 5, pp. 105-115, 1992.
8. n) by multilayer feedforward networks and related problems," IEEE Trans. Neural Netw., vol. 6, no. 1, pp. 25-30, Jan. 1995.
9. G.-B. Huang, Y.-Q. Chen, and H. A. Babri, "Classification ability of single hidden layer feedforward neural networks," IEEE Trans. Neural Netw., vol. 11, no. 3, pp. 799-801, May 2000.
10. D. Erdogmus, O. Fontenla-Romero, J. C. Principe, A. Alonso-Betanzos, and E. Castillo, "Linear-least-squares initialization of multilayer perceptrons through backpropagation of the desired response," IEEE Trans. Neural Netw., vol. 16, no. 2, pp. 325-337, Mar. 2005.
11. D. E. Rumellhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," Parallel Distrib. Process., vol. 1, pp. 318-362, 1986.
12. R. P. Brent, "Fast training algorithms for multilayer neural nets," IEEE Trans. Neural Netw., vol. 2, no. 3, pp. 346-354, May 1991.
13. R. S. Scalero and N. Tepedelenlioglu, "A fast new algorithm for training feedforward neural networks," IEEE Trans. Signal Process., vol. 40, no. 1, pp. 202-210, Jan. 1992.
14. A. Sperduti and A. Starita, "Speed up learning and network optimization with extended back propagation," Neural Netw., vol. 6, pp. 365-383, 1993.
15. A. Parlos, B. Fernandez, A. Atiya, J. Muthusami, and W. Tsai, "An accelerating learning algorithm for multilayer perceptron networks," IEEE Trans. Neural Netw., vol. 5, no. 3, pp. 493-497, May 1994.
16. X. H. Yu, G. A. Chen, and S. X. Cheng, "Dynamic learning rate optimization of the backpropagation algorithm," IEEE Trans. Neural Netw., vol. 6, no. 3, pp. 669-677, May 1995.
17. M. T. Hagan, H. B. Demuth, and M. H. Beale, Neural Network Design. Boston, MA: PWS, 1996.
18. M. T. Hagan and M. B. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 6, pp. 989-993, Nov. 1994.
19. S. Abid, F. Fnaiech, and M. Najim, "A fast feedforward training algorithm using a modified form of the standard backpropagation algorithm," IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 424-430, Mar. 2001.
20. G.-B. Huang and H. A. Babri, "Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions," IEEE Trans. Neural Netw., vol. 9, no. 1, pp. 224-229, Jan. 1998.
21. S. Ferrari and R. F. Stengel, "Smooth function approximation using neural networks," IEEE Trans. Neural Netw., vol. 16, no. 1, pp. 24-38, Jan. 2005.
22. C. Xiang, S. Q. Ding, and T. H. Lee, "Geometrical interpretation and architecture selection of mlp," IEEE Trans. Neural Netw., vol. 16, no. 1, pp. 84-96, 2005.
23. G.-B. Huang, Q.-Y. Zhu, K. Z. Mao, C.-K. Siew, P. Saratchandran, and N. Sundararajan, "Can threshold networks be trained directly?," IEEE Trans. Circuits Syst. II, vol. 53, no. 3, pp. 187-191, Mar. 2006.
24. G.-B. Huang, L. Chen, and C.-K. Siew, "Universal approximation using incremental constructive feedforward networks with random hidden nodes," IEEE Trans. Neural Netw., vol. 17, no. 4, pp. 879-892, Jul. 2006.
25. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: A new learning scheme of feedforward neural networks," in Proc. Int. Joint Conf. Neural Networks (IJCNN2004), Budapest, Hungary, Jul. 25-29, 2004, vol. 2, pp. 985-990.
26. G. B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: Theory and applications," Neurocomput., to be published.
27. C. Blake and C. Merz, UCI repository of machine learning databases Online. Available: http://www.ics.uci.edu/~mlearn/MLRepository. html Dept. Inf. Comp. Sci., Univ. California. Irvine, CA, 1998
28. D. Michie, D. J. Spiegelhalter, and C. C. Taylor, Machine Learning, Neural and Statistical Classification. New York: Ellis Horwood, 1994.
29. P. W. Frey and D. J. Slate, "Letter recognition using Holland-style adaptive classifiers," Mach. Learn., vol. 6, no. 2, pp. 161-182, 1991.
30. G.-B. Huang, K. Z. Mao, C.-K. Siew, and D.-S. Huang, "Fast modular network implementation for support vector machines," IEEE Trans. Neural Netw., vol. 16, no. 6, pp. 1651-1663, Nov. 2005.
31. R. Collobert, Y. Bengio, and S. Bengio, "Scaling large learning problems with hard parallel mixtures," Int. J. Pattern Recognit. Artif. Intell., vol. 17, no. 3, pp. 349-365, 2003.
32. R. Collobert, Y. Bengio, and S. Bengio, "A parallel mixtures of SVMs for very large scale problems," Neural Comput., vol. 14, pp. 1105-1114, 2002.