Regularization and suboptimal solutions in learning from data

Studies in Computational Intelligence

Volumn 247, Issue , 2009, Pages 113-154

  Gnecco, Giorgio ^a   Sanguineti, Marcello ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

Accuracy of suboptimal solutions; Ill posedness; Inverse problems; Regularization techniques; Weight decay

Indexed keywords

EID: 70350231420     PISSN: 1860949X     EISSN: None     Source Type: Book Series    
DOI: 10.1007/978-3-642-04003-0_6     Document Type: Article

References (99)

1. Aarts, E., Korst, J.: Simulated Annealing and Boltzmann Machines: A Stochastic Approach to Combinatorial Optimization and Neural Computing. John Wiley & Sons, Chichester (1989)
2. Alessandri, A., Sanguineti, M., Maggiore, M.: Optimization-based learning with bounded error for feedforward neural networks. IEEE Trans. on Neural Networks 13, 261-273 (2002)
3. Aronszajn, N.: Theory of reproducing kernels. Trans. of AMS 68, 337-404 (1950)
4. Aster, R., Borchers, B., Thurber, C.: Parameter Estimation and Inverse Problems. Academic Press, London (2004)
5. Barron, A.R.: Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. on Information Theory 39, 930-945 (1993)
6. Bartlett, P.L.: The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. on Information Theory 44, 525-536 (1998)
7. Berg, C., Christensen, J.P.R., Ressel, P.: Harmonic Analysis on Semigroups. Springer, New York (1984)
8. Berlinet, A., Thomas-Agnan, C.: Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer, Heidelberg (2003)
9. Bertero, M.: Linear inverse and ill-posed problems. Advances in Electronics and Electron Physics 75, 1-120 (1989)
10. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1999)
11. Bertsekas, D.P., Tsitsiklis, J.: Neuro-Dynamic Programming. Athena Scientific, Belmont (1996)
12. Bishop, C.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)
13. Bishop, C.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
14. Burger, M., Engl, H.: Training neural networks with noisy data as an ill-posed problem. Advances in Computational Mathematics 13, 335-354 (2000)
15. Burger, M., Neubauer, A.: Analysis of Tikhonov regularization for function approximation by neural networks. Neural Networks 16, 79-90 (2002)
16. Chow, T.W.S., Cho, S.Y.: Neural Networks and Computing: Learning Algorithms and Applications. World Scientific, Singapore (2007)
17. Corradi, V., White, H.: Regularized neural networks: Some convergence rate results. Neural Computation 7, 1225-1244 (1995)
18. Cortes, C., Vapnik, V.: Support vector networks. Machine Learning 20, 1-25 (1995)
19. Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)
20. Cucker, F., Smale, S.: On the mathematical foundations of learning. Bulletin of AMS 39, 1-49 (2001)
21. Cucker, F., Smale, S.: Best choices for regularization parameters in learning theory: On the bias-variance problem. Foundations of Computational Mathematics 2, 413-428 (2002)
22. Cucker, F., Zhou, D.X.: Learning Theory - An Approximation Theory Viewpoint. Cambridge University Press, Cambridge (2007)
23. Das, A., Kempe, D.: Algorithms for subset selection in linear regression. In: STOC 2008: Proc. of the 40th annual ACM symposium on Theory of computing, pp. 45-54. ACM Press, New York (2008)
24. Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximations. Constructive Approximation 13, 57-98 (1997)
25. De Mol, C., De Vito, E., Rosasco, L.: Elastic-net regularization in learning theory. J. of Complexity 25, 201-230 (2009)
26. De Vito, E., Caponnetto, A., Rosasco, L.: Discretization error analysis for Tikhonov regularization in learning theory. Analysis and Applications 4, 81-99 (2006)
27. De Vito, E., Rosasco, L., Caponnetto, A., De Giovannini, U., Odone, F.: Learning from examples as an inverse problem. J. of Machine Learning Research 6, 883-904 (2005)
28. Dontchev, A.L.: Perturbations, Approximations and Sensitivity Analysis of Optimal Control Systems. LNCIS, vol. 52. Springer, Heidelberg (1983)
29. Dunford, N., Schwartz, J.T.: Linear Operators. Part II: Spectral Theory. Interscience, New York (1963)
30. Efron, B., Hastie, T., Johnstone, L., Tibshirani, R.: Least angle regression. Annals of Statistics 32, 407-499 (2004)
31. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (2000)
32. Evgeniou, T., Pontil, M., Poggio, T.: Regularization networks and support vector machines. Advances in Computational Mathematics 13, 1-50 (2000)
33. Friedman, A.: Foundations of Modern Analysis. Dover, New York (1982)
34. Gilbert, A.C., Muthukrishnan, S., Strauss, M.J.: Approximation of functions over redundant dictionaries using coherence. In: Proc. 14th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 243-252 (2003)
35. Girosi, F.: Regularization theory, radial basis functions and networks. In: Cherkassky, V., Friedman, J.H., Wechsler, H. (eds.) From Statistics to Neural Networks. Theory and Pattern Recognition Applications. NATO ASI Series F, Computer and Systems Sciences, pp. 166-187. Springer, Berlin (1994)
36. Girosi, F.: An equivalence between sparse approximation and support vector machines. Neural Computation 10, 1455-1480 (1998)
37. Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Computation 7, 219-269 (1995)
38. Gnecco, G., Sanguineti, M.: Regularization techniques and suboptimal solutions to optimization problems in learning from data. Neural Computation (to appear)
39. Gnecco, G., Sanguineti, M.: The weight-decay technique in learning from data: An optimization point of view. Computational Management Science 6, 53-79 (2009)
40. Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading (1989)
41. Graepel, T., Herbrich, R.: From margin to sparsity. Advances in Neural Information System Processing 13, 210-216 (2001)
42. Gribonval, R., Vandergheynst, P.: On the exponential convergence of matching pursuits in quasi-incoherent dictionaries. IEEE Trans. on Information Theory 52, 255-261 (2006)
43. Grippo, L.: Convergent on-line algorithms for supervised learning in neural networks. IEEE Trans. on Neural Networks 11, 1284-1299 (2000)
44. Groetch, C.W.: Generalized Inverses of Linear Operators. Dekker, New York (1977)
45. Gupta, A., Lam, M.: The weight decay backpropagation for generalizations with missing values. Annals of Operations Research 78, 165-187 (1998)
46. Gupta, A., Lam, M.: Weight decay backpropagation for noisy data. Neural Networks 11, 1127-1138 (1998)
47. Hadamard, J.: Sur les problèmes aux dérivées partielles et leur signification physique. Bull. Univ. Princeton 13, 49-52 (1902)
48. 1-minimization: Methodology and convergence. SIAM J. on Optimization 19(3), 1107-1130 (2008)
49. Haykin, S.: Neural Networks. A Comprehensive Foundation. Macmillan, New York (1994)
50. Hofinger, A.: Nonlinear function approximation: Computing smooth solutions with an adaptive greedy algorithm. J. of Approximation Theory 143, 159-175 (2006)
51. Hofinger, A., Pillichshammer, F.: Learning a function from noisy samples at a finite sparse set of points. J. of Approximation Theory (to appear) doi:10.1016/j.jat.2008.11.003
52. Honeine, P., Richard, C., Bermudez, J.C.M.: On-line nonlinear sparse approximation of functions. In: Proc. IEEE Int. Symposium on Information Theory (ISIT), pp. 956-960. France (2007)
53. Hussain, Z., Shawe-Taylor, J.: Theory of matching pursuit. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Proc. 22nd Annual Conf. on Neural Information Processing Systems, 2008. Advances in Neural Information Processing Systems, vol. 21, pp. 721-728. MIT Press, Cambridge (2009)
54. Jones, L.K.: A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Annals of Statistics 20, 608-613 (1992)
55. Kůrková, V.: Dimension-independent rates of approximation by neural networks. In: Warwick, K., Kárný, M. (eds.) Computer-Intensive Methods in Control and Signal Processing. The Curse of Dimensionality, pp. 261-270. Birkhäuser, Boston (1997)
56. Kůrková, V.: High-dimensional approximation and optimization by neural networks. chapter 4. In: Suykens, J., et al. (eds.) Advances in Learning Theory: Methods, Models and Applications, pp. 69-88. IOS Press, Amsterdam (2003)
57. Kůrková, V.: Learning from data as an inverse problem. In: Antoch, J. (ed.) Proc. in Computational Statistics, COMPSTAT 2004, pp. 1377-1384. Physica-Verlag/Springer, Heidelberg (2004)
58. Kůrková, V., Sanguineti, M.: Bounds on rates of variable-basis and neural-network approximation. IEEE Trans. on Information Theory 47, 2659-2665 (2001)
59. Kůrková, V., Sanguineti, M.: Comparison of worst case errors in linear and neural network approximation. IEEE Trans. on Information Theory 48, 264-275 (2002)
60. Kůrková, V., Sanguineti, M.: Error estimates for approximate optimization by the extended Ritz method. SIAM J. on Optimization 15, 261-287 (2005)
61. Kůrková, V., Sanguineti, M.: Learning with generalization capability by kernel methods of bounded complexity. J. of Complexity 21, 350-367 (2005)
62. Kůrková, V., Sanguineti, M.: Approximate minimization of the regularized expected error over kernel models. Mathematics of Operations Research 33, 747-756 (2008)
63. Kůrková, V., Sanguineti, M.: Geometric upper bounds on rates of variable-basis approximation. IEEE Trans. on Information Theory 54, 5681-5688 (2008)
64. Kůrková, V., Savický, P., Hlaváčková, K.: Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Networks 11, 651-659 (1998)
65. Kainen, P.C., Kurková, V., Sanguineti, M.: Complexity of Gaussian radial-basis networks approximating smooth functions. Journal of Complexity 25, 63-74 (2009)
66. Kimeldorf, G.S., Wahba, G.: A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Annals of Mathematical Statistics 41, 495-502 (1970)
67. Kolmogorov, A.N., Fomin, S.V.: Introductory Real Analysis. Dover, New York (1970)
68. Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in Neural Information Processing Systems, vol. 4, pp. 950-957. Morgan Kaufmann, San Francisco (1992)
69. Levitin, E.S., Polyak, B.T.: Convergence of minimizing sequences in conditional extremum problems. Dokl. Akad. Nauk SSSR 168, 764-767 (1966)
70. Littlestone, N., Warmuth, M.: Relating data compression and learnability. Tech. rep., University of California, Santa Cruz (1986)
71. Lo Gerfo, L., Rosasco, L., Odone, F., De Vito, E., Verri, A.: Spectral algorithms for supervised learning. Neural Computation 20, 1873-1897 (2008)
72. Miller, K.: Least squares methods for ill-posed problems with a prescribed bound. SIAM J. of Mathematical Analysis 1, 52-74 (1970)
73. Mukherjee, S., Rifkin, R., Poggio, T.: Regression and classification with regularization. In: Denison, D., Hansen, M.H., Holmes, C.C., Mallick, B., Yu, B. (eds.) Lectures Notes in Statistics: Nonlinear Estimation and Classification - Proc. from MSRI Workshop, vol. 171, pp. 107-124. Springer, Heidelberg (2002)
74. Novikoff, A.: On convergence proofs for perceptrons. In: Proceeding of the Symposium on the Mathematical Theory of Automata, vol. 12, pp. 615-622 (1962)
75. Ortega, J.M.: Numerical Analysis: A Second Course. SIAM, Philadelphia (1990)
76. Parzen, E.: An approach to time series analysis. Annals of Mathematical Statistics 32, 951-989 (1961)
77. Pisier, G.: Remarques sur un résultat non publié de B. Maurey. In: Séminaire d'Analyse Fonctionnelle 1980-1981, vol. I(12), École Polytechnique, Centre de Mathématiques, Palaiseau, France (1981)
78. Poggio, T., Girosi, F.: Networks for approximation and learning. Proc. of the IEEE 78, 1481-1497 (1990)
79. Poggio, T., Smale, S.: The mathematics of learning: Dealing with data. Notices of the AMS 50, 536-544 (2003)
80. Roth, V.: The generalized Lasso. IEEE Trans. on Neural Networks 15, 16-28 (2004)
81. Schölkopf, B., Herbrich, R., Smola, A.J., Williamson, R.C.: A generalized representer theorem. In: Helmbold, D.P., Williamson, B. (eds.) COLT 2001. LNCS (LNAI), vol. 2111, pp. 416-424. Springer, Heidelberg (2001)
82. Schölkopf, B., Smola, A.J.: Learning With Kernels - Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002)
83. Schönberg, I.J.: Metric spaces and completely monotone functions. Annals of Mathematics 39, 811-841 (1938)
84. Serre, D.: Matrices - Theory and Applications. Springer, Heidelberg (2002)
85. Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)
86. Tibshirani, R.: Regression shrinkage and selection via the LASSO. J. of the Royal Statistical Society, Series B 58, 267-288 (1996)
87. Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-posed Problems. W.H. Winston, Washington (1977)
88. Treadgold, N.K., Gedeon, T.D.: Simulated annealing and weight decay in adaptive learning: The SARPROP algorithm. IEEE Trans. on Neural Networks 9, 662-668 (1998)
89. Tropp, J.A.: Greed is good: Algorithmic results for sparse approximation. IEEE Trans. on Information Theory 50, 2231-2242 (2004)
90. Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1998)
91. Vapnik, V.N.: Statistical Learning Theory. Wiley Interscience, New York (1998)
92. Vasin, V.V.: Relationship of several variational methods for the approximate solution of ill-posed problems. Mathematical Notes 7, 161-165 (1970)
93. Vishkin, U.: Deterministic sampling - A new technique for fast pattern matching. SIAM J. on Computing 20, 22-40 (1991)
94. Vladimirov, A.A., Nesterov, Y.E., Chekanov, Y.N.: On uniformly convex functionals. Vestnik Moskovskogo Universiteta. Seriya 15 - Vychislitel'naya Matematika i Kibernetika 3, 12-23 (1978); English translation: Moscow University Computational Mathematics and Cybernetics, 10-21 (1979)
95. Wahba, G.: Spline Models for Observational Data. Series in Applied Mathematics, vol. 59. SIAM, Philadelphia (1990)
96. Yin, G.: Rates of convergence for a class of global stochastic optimization algorithms. SIAM J. on Optimization 10, 99-120 (1999)
97. Zhang, T.: Approximation bounds for some sparse kernel regression algorithms. Neural Computation 14, 3013-3042 (2002)
98. Zhang, T.: Sequential greedy approximation for certain convex optimization problems. IEEE Trans. on Information Theory 49(3), 682-691 (2003)
99. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. of the Royal Statistical Society B 67, 301-320 (2005)