Regularization techniques and suboptimal solutions to optimization problems in learning from data

Neural Computation

Volumn 22, Issue 3, 2010, Pages 793-829

  Gnecco, Giorgio ^a   Sanguineti, Marcello ^b  

^a UNIVERSITY OF GENOA   (Italy)

^b Department of Communications Computer and System Sciences ^*  (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; COMPARATIVE STUDY; STATISTICAL MODEL; TIME;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; LINEAR MODELS; TIME FACTORS;

EID: 77952978333     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2009.05-08-786     Document Type: Article

References (59)

1. Aronszajn, N. (1950). Theory of reproducing kernels. Trans. of AMS, 68, 337-404.
2. Bachmann, G., & Narici, L. (2000). Functional analysis. New York: Dover.
3. Bartlett, P. L., & Mendelson, S. (2002). Rademacher and gaussian complexities: Risk bounds and structural results. J. Machine Learning Research, 3, 463-482.
4. Berg, C., Christensen, J. P. R., & Ressel, P. (1984). Harmonic analysis on semigroups. New York: Springer
5. Bertero, M. (1989). Linear inverse and ill-posed problems. Advances in Electronics and Electron Physics, 75, 1-120.
6. Bousquet, O., & Elisseeff, A. (2002). Stability and generalization. J. Machine Learning Research, 2, 499-526.
7. Cucker, F., & Smale, S. (2001). On the mathematical foundations of learning. Bulletin of AMS, 39, 1-49.
8. Cucker, F., & Smale, S. (2002). Best choices for regularization parameters in learning theory: On the bias-variance problem. Foundations of Computational Mathematics, 2, 413-428.
9. Das, A., & Kempe, D. (2008). Algorithms for subset selection in linear regression. In Proc. 40th Annual ACM Symposium on Theory of Computing (pp. 45-54). New York: ACM.
10. Davis, G., Mallat, S., & Avellaneda, M. (1997). Adaptive greedy approximations. Constructive Approximation, 13, 57-98.
11. De Mol, C., De Vito, E., & Rosasco, L. (2009). Elastic-net regularization in learning theory. J. Complexity, 25, 201-230.
12. De Vito, E., Rosasco, L., Caponnetto, A., De Giovannini, U., & Odone, F. (2005). Learning from examples as an inverse problem. J. Machine Learning Research, 6, 883-904.
13. Dontchev,A. L.,&Zolezzi, T. (1993).Well-posed optimization problems. Berlin: Springer.
14. Drineas, P., & Mahoney, M.W. (2005). On the Nystr̈om method for approximating a Gram matrix for improved kernel-based learning. J. Machine Learning Research, 6, 2153-2175.
15. Efron, B., Hastie, T., Johnstone, L., & Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32, 407-499.
16. Engl, H. W., Hanke, M., & Neubauer, A. (2000). Regularization of inverse problems. Norwood, MA: Kluwer.
17. Evgeniou, T., Pontil, M., & Poggio, T. (2000). Regularization networks and support vector machines. Advances in Computational Mathematics, 13, 1-50.
18. Friedman, A. (1982). Foundations of modern analysis. New York: Dover.
19. Gilbert, A. C.,Muthukrishnan, S.,&Strauss, M. J. (2003).Approximation of functions over redundant dictionaries using coherence. In Proc. 14th Annual ACM-SIAM Symposium on Discrete Algorithms (pp. 243-252). Philadelphia: SIAM.
20. Girosi, F. (1994). Regularization theory, radial basis functions and networks. In V. Cherkassky, J. H. Friedman, & H. Wechsler (Eds.), From statistics to neural networks: Theory and pattern recognition applications (pp. 166-187). Berlin: Springer.
21. Girosi, F. (1998). An equivalence between sparse approximation and support vector machines. Neural Computation, 10, 1455-1480.
22. Gribonval, R.,&Vandergheynst, P. (2006).On the exponential convergence of matching pursuits in quasi-incoherent dictionaries. IEEE Trans. on Information Theory, 52, 255-261.
23. Groetch, C.W. (1977). Generalized inverses of linear operators. NewYork: Dekker.
24. 1-minimization: Methodology and convergence. SIAM J. Optimization, 19, 1107-1130.
25. Honeine, P., Richard, C., & Bermudez, J. C. M. (2007). On-line nonlinear sparse approximation of functions. In Proc. IEEE Int. Symposium on Information Theory (pp. 956-960). Piscataway, NJ: IEEE Press.
26. Hussain, Z., & Shawe-Taylor, J. (2009). Theory of matching pursuit. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (Eds.), Advances in neural information processing systems, 21. Cambridge, MA: MIT Press.
27. Kolmogorov, A. N., & Fomin, S. V. (1975). Introductory real analysis.New York: Dover.
28. Kůrková, V. (1997). Dimension-independent rates of approximation by neural networks. In K. Warwick & M. Kárný (Eds.), Computer-intensive methods in control and signal processing: The curse of dimensionality (pp. 261-270). Boston: Birkḧauser.
29. K urkov́a, V. (2004). Learning from data as an inverse problem. In J. Antoch (Ed.), Proc. Int. Conf. on Computational Statistics (pp. 1377-1384). Heidelberg: Physica/ Springer.
30. Kůrková, V. (2005). Neural network learning as an inverse problem. Logic Journal of IGPL, 13, 551-559.
31. Kůrková, V., & Neruda, R. (1994). Uniqueness of functional representations by gaussian basis function networks. In Proc. ICANN'94 (pp. 471-474). New York: Springer.
32. Kůrková, V., & Sanguineti, M. (2001). Bounds on rates of variable-basis and neuralnetwork approximation. IEEE Trans. on Information Theory, 47, 2659-2665.
33. Kůrková, V., & Sanguineti, M. (2002). Comparison of worst case errors in linear and neural network approximation. IEEE Trans. on Information Theory, 48, 264-275.
34. Kůrková, V., & Sanguineti,M. (2005a). Error estimates for approximate optimization by the extended Ritz method. SIAM J. Optimization, 15, 461-487.
35. Kůrková, V., & Sanguineti, M. (2005b). Learning with generalization capability by kernel methods of bounded complexity. J. Complexity, 21, 350-367.
36. Limaye, B. V. (1996). Functional analysis. New Delhi: New Age Publishers.
37. Makovoz, Y. (1996). Random approximants and neural networks. J. Approximation Theory, 85, 98-109.
38. Mendelson, S. (2003). A few notes on statistical learning theory. In S. Mendelson & A. Smola (Eds.), Lecture notes in computer science (pp. 1-40). New York: Springer.
39. Miller, K. (1970). Least squares methods for ill-posed problems with a prescribed bound. SIAM J. Mathematical Analysis, 1, 52-74.
40. Mukherjee, S., Rifkin, R., & Poggio, T. (2002). Regression and classification with regularization. In D. Denison, M. H. Hansen, C. C. Holmes, B. Mallick, and B. Yu (Eds.), Lectures notes in statistics: Nonlinear estimation and classification (Proc. MSRI Workshop) (pp. 107-124). New York: Springer.
41. Ortega, J. M. (1990). Numerical analysis: A second course. Philadelphia: SIAM. Rauch, J. (1991). Partial differential equations. NewYork: Springer.
42. Rauch, J. (1991). Partial differential equations. NewYork: Springer.
43. Roth, V. (2004). The generalized Lasso. IEEE Trans. on Neural Networks, 15, 16-28.
44. Scḧolkopf, B., & Smola, A. J. (2002). Learning with kernels-Support vector machines, regularization, optimization, and beyond. Cambridge, MA: MIT Press.
45. Serre, D. (2002). Matrices-Theory and applications. NewYork: Springer.
46. Shawe-Taylor, J., Bartlett, P. L., Williamson, R. C., & Anthony, M. (1998). Structural risk minimization over data-dependent hierarchies. IEEE Trans. on Information Theory, 44, 1926-1940.
47. Shawe-Taylor, J., & Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge: Cambridge University Press.
48. Smola, A. J.,&Scḧolkopf, B. (2000). Sparse greedymatrix approximation formachine learning. In Proc. 17th Int. Conf. on Machine Learning (pp. 911-918). San Francisco: Morgan Kaufmann.
49. Tibshirani, R. (1996). Regression shrinkage and selection via the LASSO. J. Royal Statistical Society, Series B, 58, 267-288.
50. Tikhonov, A. N., & Arsenin, V. Y. (1977). Solutions of ill-posed problems. Washington, DC: Winston.
51. Tropp, J. A. (2004). Greed is good: Algorithmic results for sparse approximation. IEEE Trans. on Information Theory, 50, 2231-2242.
52. Vapnik, V. N. (1998). Statistical learning theory. New York:Wiley.
53. Vasin, V. V. (1970). Relationship of several variational methods for the approximate solution of ill-posed problems. Mathematical Notes, 7, 161-165.
54. Vincent, P., & Bengio, Y. (2002). Kernel matching pursuit. Machine Learning, 48, 165-187.
55. Wahba, G. (1990). Spline models for observational data. Philadelphia: SIAM.
56. Zhang, T. (2002a). Approximation bounds for some sparse kernel regression algorithms. Neural Computation, 14, 3013-3042.
57. Zhang, T. (2002b). A general greedy approximation algorithm with applications. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems, 14. Cambridge, MA: MIT Press.
58. Zhang, T. (2003). Sequential greedy approximation for certain convex optimization problems. IEEE Trans. on Information Theory, 49, 682-691.
59. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. J. Royal Statistical Society B, 67, 301-320.