Universal approximation with convex optimization: Gimmick or reality? [Discussion Forum]

IEEE Computational Intelligence Magazine

Volumn 10, Issue 2, 2015, Pages 68-77

  Principe, Jose C ^a   Chen, Badong ^b  

^a J Crayton Pruitt Family Department of Biomedical Engineering   (United States)

^b XI'AN JIAOTONG UNIVERSITY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CONVEX OPTIMIZATION; VECTOR SPACES;

DISCUSSION FORUM; ECHO STATE NETWORKS; EXTREME LEARNING MACHINE; KERNEL ADAPTIVE FILTERS; LEARNING MACHINES; MULTI LAYER PERCEPTRON; OPTIMIZATION METHOD; UNIVERSAL APPROXIMATION;

LEARNING SYSTEMS;

EID: 84928043061     PISSN: 1556603X     EISSN: None     Source Type: Journal    
DOI: 10.1109/MCI.2015.2405352     Document Type: Review

References (48)

1. J. Principe, N. Euliano, and C. Lefebvre, Neural and Adaptive Systems: Fundamentals Through Simulations. Hoboken, NJ: Wiley, 2000
2. C. M. Bishop, Neural Networks for Pattern Recognition. New York: Oxford Univ. Press, 1995
3. D. Hilbert, "Mathematische probleme," Nachr. Akad. Wiss. Gottingen, pp. 290-329, 1900
4. N. Kolmogorov, "On the representation of continuous functions of several variables by superposition of continuous functions of one variable and addition," Dokl. Akad. Nauk SSSR, vol. 114, no. 5, pp. 953-956, 1957
5. F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Washington, D. C.: Spartan Books, 1962
6. M. Minsky and S. Papert, Perceptrons. Cambridge, MA: MIT Press, 1969
7. N. Nilsson, The Mathematical Foundations of Learning Machines. San Francisco, CA: Morgan Kaufmann, 1965
8. F. Girosi and T. Poggio, "Networks for learning: A view from the theory of approximation of functions," in Neural Networks: Concepts, Applications, and Implementations, vol. I, P. Antognetti and V. Milutinovic, Eds. Englewood Cliffs, NJ: Prentice Hall, 1991, no. 6, pp. 110-155
9. G. Cybenko, "Approximation by superposition of a sigmoidal function," Math. Control Syst. Signals, vol. 2, no. 4, pp. 303-314, 1989
10. T. M. Cover, "Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition," IEEE Trans. Electron. Comput., vol. 14, no. 3, pp. 326-334, 1965
11. A. N. Kolmogorov, "Stationary sequences in Hilbert space," (In Russian) Bull. Moscow Univ. vol. 2 no. 6, pp. 1-40, 1941. English translation in Linear Least Squares Estimation, T. Kailath, Ed. Stroudsburg, PA: Dowden, Hutchinson &Ross, 1977
12. P. Werbos, "Beyond regression: New tools for prediction and analysis in the behavioral sciences," Ph.D. dissertation, Harvard Univ., Cambridge, MA, 1974
13. D. Rumelhart, G. Hinton, and R. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing. Cambridge, MA: MIT Press, 1986, ch. 8, pp. 318-362
14. Y. LeCun, "Une procedure d'apprentissage pour reseau a seuila symmetrique (A learning scheme for asymmetric threshold networks)," in Proc. Cognitiva, Paris, France, 1985, vol. 85, pp. 599-604
15. G. Orr and K.-R. Muller, Eds., Neural Networks: Tricks of the Trade Springer. Berlin Heidelberg, Germany: Springer-Verlag, 1998
16. S. Haykin, Neural Networks: A Comprehensive Foundation. London, U.K.: MacMillan, 1995
17. X. Yao, "Evolutionary artificial neural networks," Int. J. Neural Syst., vol. 4, no. 3, pp. 203-222, 1993
18. B. Igelnik and Y.-H. Pao, "Stochastic choice of basis functions in adaptive function approximation and the functional-link net," IEEE Trans. Neural Netw., vol. 6, no. 6, pp. 1320-1329, Nov. 1995
19. N. Wiener, Nonlinear Problems in Random Theory. Cambridge, MA and New York: MIT Press and Wiley, 1958
20. S.-K. Oh, W. Pedryc, and B.-J. Park, "Polynomial neural networks architecture: Analysis and design," Comput. Electr. Eng., vol. 29, no. 6, pp. 703-725, Aug. 2003
21. V. Vapnik, The Nature of Statistical Learning Theory. Berlin Heidelberg, Germany: Springer-Verlag, 1999
22. H. Jaeger, "The echo state approach to analyzing and training recurrent neural networks," German Natl. Res. Center Inform. Tech., Tech. Rep. GMD Report 148, 2001
23. W. Maass, "Liquid state machines: Motivation, theory, and applications," in Computability in Context: Computation and Logic in the Real World, B. Cooper and A. Sorbi, Eds. London, U.K.: Imperial College Press, 2010, pp. 275-296
24. H. Jaeger, W. Maass, and J. Principe, "Special issue on echo state networks and liquid state machines," Neural Netw., vol. 20, no. 3, pp. 287-289, 2007
25. I. Sandberg and L. Xu, "Uniform approximation of multidimensional myopic maps," IEEE Trans. Circuits Syst. I, Fundam. Theory Applicat., vol. 44, no. 6, pp. 477-500, 1997
26. M. Ozturk, D. Xu, and J. Principe, "Analysis and design of echo state networks for function approximation," Neural Comput., vol. 19, no. 1, pp. 111-138, 2007
27. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: A new learning scheme of feedforward neural networks," in Proc. Int. Joint Conf. Neural Networks, 2004, vol. 2, pp. 985-990
28. T. Tao and V. Vu, "Random matrices: Universal properties of eigenvectors," Random Matrices: Theory, Applicat., 2012
29. G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, "Extreme learning machine: Theory and applications," Neurocomputing, vol. 70, nos. 1-3, pp. 489-501, 2006
30. G.-B. Huang, "An insight into extreme learning machines: Random neurons, random features and kernels," Cogn. Comput., vol. 6, no. 3, pp. 376-390, 2014
31. G.-B. Huang, S. Song, and K. You, "Trends in extreme learning machines: A review," Neural Netw., vol. 61, pp. 32-48, Jan. 2015
32. W. Liu, J. Principe, and S. Haykin, Kernel Adaptive Filtering: A Comprehensive Introduction. Hoboken, NJ: Wiley, 2010
33. W. Liu, P. Pokarel, and J. Principe, "The kernel LMS algorithm," IEEE Trans. Signal Processing, vol. 56, no. 2, pp. 543-554, 2008
34. D. Broomhead and D. Lowe, "Multivariable functional interpolation and adaptive networks," Complex Syst., vol. 2, pp. 321-355, 1988
35. N. Aronszajn, "Theory of reproducing kernels," Trans. Amer. Math. Soc., vol. 68, no. 3, pp. 337-404, 1950
36. B. Widrow, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice Hall, 1985
37. G. Wahba, Spline Models for Observational Data. Philadelphia, PA: SIAM, 1990
38. B. Chen, S. Zhao, P. Zhu, and J. Principe, "Quantized kernel least mean square algorithm," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 1, pp. 22-32, 2012
39. S. Zhao, B. Chen, P. Zhu, and J. Principe, "Fixed budget quantized kernel least-mean-square algorithm," Signal Process., vol. 93, no. 9, pp. 2759-2770, Sept. 2013
40. B. Chen, S. Zhao, P. Zhu, and J. Principe, "Quantized kernel recursive least squares algorithm," IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 9, pp. 1484-1491, 2013
41. B. Chen, S. Zhao, P. Zhu, and J. Principe, "Mean square convergence analysis for kernel least mean square algorithm," Signal Process., vol. 92, no. 11, pp. 2624-2632, 2012
42. W. Liu, I. Park, Y. Wang, and J. Principe, "Extended kernel recursive least squares algorithm," IEEE Trans. Signal Processing, vol. 57, no. 10, pp. 3801-3814, 2009
43. I. Park, S. Seth, M. Rao, and J. Principe, "Strictly positive def inite kernels for point process divergences," Neural Comput., vol. 24, no. 8, pp. 2223-2250, Aug. 2012
44. C. Chen, "A rapid supervised learning neural network for function interpolation and approximation," IEEE Trans. Neural Netw., vol. 7, no. 5, pp. 1220-1230, 1996
45. G.-B. Huang, H. Zhou, X. Ding, and R. Zhang, "Extreme learning machine for regression and multiclass classification," IEEE Trans. Syst. Man, Cybern. B, vol. 42, no. 2, pp. 513-529, 2012
46. R. Pokharel, S. Seth, and J. Principe, "Quantized mixture kernel least mean square," in Proc. IEEE World Congr. Computational Intelligence, Beijing, China, 2014
47. B. Chen, J. Liang, N. Zheng, and J. Principe, "Kernel least mean square with adaptive kernel size," ArXiv preprint arXiv: 1401.5899, 2014
48. P. Zhu, B. Chen, and J. Principe, "Learning nonlinear generative models of time series with a Kalman filter in RKHS," IEEE Trans. Signal Processing, vol. 62, no. 1, pp. 141-155, 2014