Approximate minimization of the regularized expected error over kernel models

Mathematics of Operations Research

Volumn 33, Issue 3, 2008, Pages 747-756

  Kůrková, Věra ^a   Sanguineti, Marcello ^b  

^a INSTITUTE OF COMPUTER SCIENCE   (Czech Republic)

^b UNIVERSITY OF GENOA   (Italy)

Author keywords

Convex functionals; Expected error; Kernel methods; Model complexity; Rates of convergence; Suboptimal solutions

Indexed keywords

CONVEX FUNCTIONALS; EXPECTED ERROR; KERNEL METHODS; MODEL COMPLEXITY; RATES OF CONVERGENCE; SUBOPTIMAL SOLUTIONS;

OPTIMIZATION;

EID: 61349203972     PISSN: 0364765X     EISSN: 15265471     Source Type: Journal    
DOI: 10.1287/moor.1080.0317     Document Type: Article

References (41)

1. Aarts, E., J. Korst. 1989. Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing. John Wiley & Sons, New York.
2. Aizerman, M. A., E. M. Braverman, L. I. Rozonoer. 1964. Theoretical foundations of the potential function method in pattern recognition learning. Automation Remote Control 25 821-837.
3. Alessandri, A., M. Sanguineti, M. Maggiore. 2002. Optimization-based learning with bounded error for feedforward neural networks. IEEE Trans. Neural Networks 13(2) 261-273.
4. Aronszajn, N. 1950. Theory of reproducing kernels. Trans. Amer: Math. Soc. 68(3) 337-404.
5. Barron, A. R. 1993. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39(33) 930-945.
6. Berg, C., J. P. R. Christensen, P. Ressel. 1984. Harmonic Analysis on Semigroups. Springer-Verlag, New York.
7. Berlinet, A., C. Thomas-Agnan. 2004. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer.
8. Bertero, M. 1989. Linear inverse and ill-posed problems. Adv. Electronics Electron Phys. 75 1-120.
9. Bertsekas, D. P. 1999. Nonlinear Programming. Athena Scientific, Belmont, MA.
10. Bertsekas, D. P., J. N. Tsitsiklis. 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA.
11. Borwein, J., A. Lewis. 2000. Convex Analysis and Nonlinear Optimization. CMS Books in Mathematics. Springer, New York.
12. Burger, M., A. Neubauer. 2003. Analysis of Tikhonov regularization for function approximation by neural networks. Neural Networks 16 79-90.
13. Chen, S. S., D. L. Donoho, M. A. Saunders. 1998. Atomic decomposition by Basis Pursuit. SIAM J. Sci. Comput. 20 33-61.
14. Cortes, C., V. Vapnik. 1995. Support-vector networks. Machine Learn. 20 273-297.
15. Cucker, F., S. Smale. 2002. On the mathematical foundations of learning. Bull. AMS 39 1-49.
16. Dontchev, A. L. 1983. Perturbations, Approximations and Sensitivity Analysis of Optimal Control Systems. Lecture Notes in Control and Information Sciences, Vol. 52. Springer-Verlag, Berlin.
17. Dunford, N., J. T. Schwartz. 1963. Linear Operators. Part II: Spectral Theory. Interscience Publishers, New York.
18. Fine, T. L. 1999. Feedforward Neural Network Methodology. Springer.
19. Friedman, A. 1982. Modern Analysis. Dover, New York.
20. Girosi, F. 1998. An equivalence between sparse approximation and support vector machines. Neural Comput. 10 1455-1480.
21. Girosi, F., M. Jones, T. Poggio. 1995. Regularization theory and neural networks architectures. Neural Comput. 7 219-269.
22. Goldberg, D. E. 1989. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley,
23. Grippo, L. 2000. Convergent on-line algorithms for supervised learning in neural networks. IEEE Trans. Neural Networks 11(6) 1284-1299.
24. Jones, L. K. 1992. A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Ann. Statist. 20(1) 608-613.
25. Kimeldorf, G. S., G. Wahba. 1970. A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist. 41(2) 495-502.
26. Kimeldorf, G. S., G. Wahba. 1971. Some results on Tchebyceffian spline functions. J. Math. Anal. Appl. 33(1) 82-95.
27. Kůrková, V., M. Sanguineti. 2002. Comparison of worst-case errors in linear and neural network approximation. IEEE Trans. Inform. Theory 48(1) 264-275.
28. Kůrková, V., M. Sanguineti. 2005a. Error estimates for approximate optimization by the extended Ritz method. SIAM J. Optim. 15(2) 261-287.
29. Kůrková, V., M. Sanguineti. 2005b. Learning with generalization capability by kernel methods of bounded complexity. J. Complexity 21 350-367.
30. Kůrková, V., P. C. Kainen, V. Kreinovich. 1997. Estimates of the number of hidden units and variation with respect to half-spaces. Neural Networks 10(6) 1061-1068.
31. Parzen, E. 1961. An approach to time series analysis. Ann. Math. Statist. 32 951-989.
32. Parzen, E. 2006. Reproducing kernel hilbert spaces outline. http://www.stat.tamu.edu/~jianhua/rkhs/Manny.pdf.
33. Pisier, G. 1980-81. Remarques sur un resultat non publié de B. Maurey. Seminaire d'Analyse Fonctionelle I(12). École Polytechnique, Centre de Mathématiques, Palaiseau, France.
34. Poggio, T., F. Girosi. 1990. Networks for approximation and learning. Proc. IEEE 78(9) 1481-1497.
35. Poggio, T., S. Smale. 2003. The mathematics of learning: Dealing with data. Notices of the AMS 50(5) 537-544.
36. Schölkopf, B., A. J. Smola. 2002. Learning with Kernels. The MIT Press, Cambridge, MA.
37. Schönberg, I. J. 1938. Metric spaces and completely monotone functions. Ann. Math. 39(4) 811-841.
38. Tikhonov, A. N. 1963. Solutions of incorrectly formulated problems and the regularization method. Soviet Math. Dokl. 4 1035-1038.
39. Vapnik, V. N. 1998. Statistical Learning Theory. John Wiley & Sons, New York.
40. Wahba, G. 1990. Splines models for observational data. CBMS-NSF Regional Conference Series in Applied Mathematics 59. SIAM, Philadelphia.
41. Yin, G. 1963. Rates of convergence for a class of global stochastic optimization algorithms. SIAM J. Optim. 10 99-120.