A new Jacobian matrix for optimal learning of single-layer neural networks

IEEE Transactions on Neural Networks

Volumn 19, Issue 1, 2008, Pages 119-129

  Peng, Jian Xun ^a   Li, Kang ^a   Irwin, George W ^a  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

Author keywords

Approximation; Convergence; Jacobian matrix; Single hidden layer feedforward neural network (SLFN); Supervised learning; Training accuracy

Indexed keywords

APPROXIMATION THEORY; COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; JACOBIAN MATRICES; LEARNING ALGORITHMS; OPTIMIZATION; SUPERVISED LEARNING;

LEVENBERG-MARQUARDT ALGORITHM; SINGLE-HIDDEN-LAYER FEEDFORWARD NEURAL NETWORK;

FEEDFORWARD NEURAL NETWORKS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; HUMAN; LEARNING; NONLINEAR SYSTEM; PHYSIOLOGY;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; HUMANS; LEARNING; NEURAL NETWORKS (COMPUTER); NONLINEAR DYNAMICS;

EID: 39549096279     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2007.903150     Document Type: Article

References (46)

1. J. Park and I. W. Sandberg, "Universal approximation using radial-basis-function networks," Neural Comput., vol. 3, pp. 246-257, 1991.
2. K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximator," Neural Netw., vol. 2, pp. 359-366, 1989.
3. R. M. Sanner and J. E. Slotine, "Gaussian networks for direct adaptive control," IEEE Trans. Neural Netw., vol. 3, no. 6, pp. 837-863, Nov. 1992.
4. S. S. Ge, C. C. Hang, T. H. Lee, and T. Zhang, Stable Adaptive Neural Network Control. Boston, MA: Kluwer, 2001.
5. S. Chen, S. A. Billings, and W. Luo, "Orthogonal least squares methods and their application to non-linear system identification," in Int. J. Control, 1989, vol. 50, pp. 1873-1896.
6. S. Huang, K. Tan, and T. Lee, "Nonlinear adaptive control of interconnected systems using neural networks," IEEE Trans. Neural Netw., vol. 17, no. 1, pp. 243-246, Jan. 2006.
7. B. Lin, B. Lin, F. Chong, and F. Lai, "Higher-order-statistics-based radial basis function networks for signal enhancement," IEEE Trans. Neural Netw., vol. 18, no. 3, pp. 823-832, May 2007.
8. A. Vemuri, M. Polycarpou, and S. Diakourtis, "Neural network based fault detection and accommodation in robotic manipulators," IEEE Trans. Robot. Autom., vol. 14, no. 2, pp. 342-348, Apr. 1998.
9. S. Liu and J. Wang, "A simplified dual neural network for quadratic programming with its KWTA application," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1500-1510, Nov. 2006.
10. F. L. Lewis and T. Parisini, "Guest editorial: Neural network feedback with guaranteed stability," Int. J. Control, vol. 70, no. 3, pp. 337-339, 1998.
11. J.-T. Tsai, J.-H. Chou, and T.-K. Liu, "Tuning the structure and parameters of a neural network by using hybrid Taguchi-genetic algorithm," IEEE Trans. Neural Netw., vol. 17, no. 1, pp. 69-80, Jan. 2006.
12. N. Xie and H. Leung, "Blind equalization using a predictive radial basis function neural network," IEEE Trans. Neural Netw., vol. 16, pp. 709-720, May 2005.
13. H. Leung, T. Lo, and S. Wang, "Prediction of noisy chaotic time series using an optimal radial basis function neural network," IEEE Trans. Neural Netw., vol. 12, no. 5, pp. 1163-1172, Sep. 2001.
14. C. C. Min, D. Srinivasan, and R. Cheu, "Neural networks for continuous online learning and control," IEEE Trans. Neural Netw., vol. 17, no. 5, pp. 1511-1531, Nov. 2006.
15. A. Zhu and S. Yang, "A neural network approach to dynamic task assignment of multirobots," IEEE Trans. Neural Netw., vol. 17, no. 5, pp. 1278-1287, Sep. 2006.
16. G.-B. Huang, P. Saratchandran, and N. Sundararajan, "A generalized growing and pruning RBF (ggap-RBF) neural network for function approximation," IEEE Trans. Neural Netw., vol. 16, no. 1, pp. 57-67, Jan. 2005.
17. J. Peng, K. Li, and D. S. Huang, "A hybrid forward algorithm for RBF neural network construction," IEEE Trans. Neural Netw., vol. 17, no. 6, pp. 1439-1451, Nov. 2006.
18. G.-B. Huang, Q.-Y. Zhu, K. Z. Mao, C.-K. Siew, P. Saratchandran, and N. Sundararajan, "Can threshold networks be trained directly?," IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 53, no. 3, pp. 187-191, Mar. 2006.
19. K. M. Adeney and M. J. Korenberg, "Iterative fast orthogonal search algorithm for MDL-based training of generalized single-layer network," Neural Netw., vol. 13, pp. 787-799, 2000.
20. R. Battiti, "First and second-order methods for learning: Between steepest descent and Newton's method," Neural Comput., vol. 4, no. 2, pp. 141-166, 1992.
21. R. Reed, "Pruning algorithms - A survey," IEEE Trans. Neural Netw. vol. 4, no. 5, pp. 740-747, Sep. 1993.
22. N. B. Karayiannis, "Reformulated radial basis neural networks trained by gradient descent," IEEE Trans. Neural Netw., vol. 10, no. 3, pp. 657-671, May 1999.
23. R. Neruda and P. Kudova, "Learning methods for radial basis function networks," Future Generation Comput. Syst., vol. 21, pp. 1131-1142, 2005.
24. K. Levenberg, "A method for the solution of certain problems in least squares," Quart. Appl. Math., vol. 54, pp. 164-168, 1944.
25. D. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters," SIAM J. Appl. Math., vol. 11, pp. 431-441, 1963.
26. J. J. More, "The Levenberg-Marquardt algorithm: Implementation and theory," in Lecture Notes in Mathematics. New York: Springer-Verlag, 1978, pp. 105-116.
27. D. Gorinevsky, "An approach to parametric nonlinear least square optimization and application to task-level learning control," IEEE Trans. Autom. Control, vol. 42, no. 7, pp. 912-927, Jul. 1997.
28. M. T. Hagan and M. B. Menhaj, "Training feedforward networks with the Marquardt algorithm," IEEE Trans. Neural Netw., vol. 5, no. 5, pp. 989-993, Sep. 1994.
29. S. Kollias and D. Anastassiou, "An adaptive least squares algorithm for the efficient training of multilayered networks," IEEE Trans. Circuit Syst., vol. 36, no. 8, pp. 1092-1101, Aug. 1989.
30. N. Ampazis and S. J. Perantonis, "Two highly efficient second-order algorithms for training feedforward networks," IEEE Trans. Neural Netw., vol. 13, no. 5, pp. 1064-1074, Sep. 2002.
31. H. Peng, T. Ozaki, V. Haggan-Ozaki, and Y. Toyoda, "A parameter optimization method for radial basis function type models," IEEE Trans. Neural Netw., vol. 14, no. 2, pp. 432-438, Mar. 2003.
32. K. Li, J. Peng, and G. W. Irwin, "A fast nonlinear model identification method," IEEE Trans. Autom. Control, vol. 50, no. 8, pp. 1211-1216, Aug. 2005.
33. K. Li, J. Peng, and E. W. Bai, "A two-stage algorithm for identification of nonlinear dynamic systems," Automatica, vol. 42, no. 7, pp. 1189-1197, 2006.
34. S. McLoone, M. D. Brown, G. W. Irwin, and G. Lightbody, "A hybrid linear/nonlinear training algorithm for feedforward neural networks," IEEE Trans. Neural Netw., vol. 9, no. 4, pp. 669-684, Jul. 1998.
35. S. Chen, S. A. Billings, and W. Luo, "Orthogonal least squares methods and their application to non-linear system identification," Int. J. Control, vol. 50, no. 5, pp. 1873-1896, 1989.
36. G.-B. Huang, Q.-Y. Zhu, K. Z. Mao, C.-K. Siew, P. Saratchandran, and N. Sundararajan, "Can threshold networks be trained directly?," IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 53, no. 3, pp. 187-191, Mar. 2006.
37. E. L. Sutanto, J. D. Mason, and K. Warwick, "Mean-tracking clustering algorithm for radial basis function centre selection," Int. J. Control, vol. 67, pp. 961-977, 1997.
38. N. Benoudjit, C. Archambeau, A. Lendasse, J. Lee 1, and M. Verleysen, "Width optimization of the Gaussian kernels in radial basis function networks," in Proc. Eur. Symp. Artif. Neural Netw., Apr. 2002, pp. 425-432.
39. Z. Uykan, C. Guzelis, M. E. Celebi, and H. N. Koivo, "Analysis of input-output clustering for determining centers of RBFN," IEEE Trans. Neural Netw., vol. 11, no. 4, pp. 851-858, Jul. 2000.
40. Y. Shang and B. W. Wah, "Global optimization for neural network training," Computer, vol. 29, no. 3, pp. 45-54, Mar. 1996.
41. G. Bilbro, "Fast stochastic global optimization," IEEE Trans. Syst. Man, Cybern., vol. 24, no. 4, pp. 684-689, Apr. 1994.
42. V. P. Plagianakos, G. D. Magoulas, and M. Vrahatis, "Deterministic non-monotone strategies for effective training of multilayer perceptrons," IEEE Trans. Neural Netw., vol. 13, no. 6, pp. 1268-1284, Nov. 2002.
43. M. Bianchini, M. Gori, and M. Maggini, "On the problem of local minima in recurrent neural networks," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 167-177, Mar. 1994.
44. M. K. Muezzinoglu, C. Guzelis, and J. Zurada, "An energy function-based design method for discrete Hopfield associative memory with attractive fixed points," IEEE Trans. Neural Netw., vol. 16, no. 2, pp. 370-378, Mar. 2005.
45. D. Wedge, D. Ingram, D. McLean, M. C. Mingham, and Z. Bandar, "On global-local artificial neural networks for function approximation," IEEE Trans. Neural Netw., vol. 17, no. 4, pp. 942-952, Jul. 2006.
46. P. P. Palmes, T. Hayasaka, and S. Usui, "Mutation-based genetic neural network," IEEE Trans. Neural Netw., vol. 16, no. 3, pp. 587-600, May 2005.