Additive regularization trade-off: Fusion of training and validation levels in kernel methods

Machine Learning

Volumn 62, Issue 3, 2006, Pages 217-252

  Pelckmans, K ^a   Suykens, J A K ^a   De Moor, B ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Least Squares Support Vector Machines; Model selection; Optimizatioon; Regulatization

Indexed keywords

DATABASE SYSTEMS; MATHEMATICAL MODELS; OPTIMIZATION; PROBLEM SOLVING;

LEAST SQUARES SUPPORT VECTOR MACHINES; MODEL SELECTION; REGULATIZATION;

LEARNING SYSTEMS;

EID: 33644990982     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-005-5315-x     Document Type: Article

References (48)

1. Akaike, H. (1973). Statistical predictor identification. Ann. Inst. Statist. Math, 22, 203-217.
2. Bertero, M., Poggio, T. A., & Torre, T. (1988). Ill-posed problems in early vision. Proceedings of the IEEE, 76(8), 869-889.
3. Bishop, C. M. (1995). Neural Networks for Pattern Recognition. Oxford University Press.
4. Boyd, S., Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.
5. Bousquet, O., & Elisseeff, A. (2002). Stability and generalisation. Journal of Machine Learning Research, 2, 499-526.
6. Herrmann, D. J. L. & Bousquet, O. (2003). On the complexity of learning the kernel matrix. In Becker, S., Thrun, S. & Obermayer, K. (Eds.), Advances in Neural Information Processing Systems MIT Press, Cambridge, MA, (pp. 399-406).
7. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123-140.
8. Burman, P. (1989). A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika, 76(3), 503-514.
9. Cawley, G. C., & Talbot, N.L.C (2003). Efficient leave-one-out cross-validation of kernel Fisher discriminant classifiers. Pattern Recognition, 56(11), 2585-2592.
10. Chapelle, O., & Vapnik, V. (2000). Model selection for support vector machines. Advances in Neural Information Processing Systems, S. A. Solla, T. K. Leen & K.-R. Muller, (Eds) MIT Press, 12.
11. Chapelle, O., Vapnik, V., Bousquet, O., & Mukherjee, S. (2002). Choosing multiple parameters for support vector machines. Machine Learning, 46(1-3), 131-159.
12. Cherkassky, V., & Mulier, F. (1998). Learning from Data. Wiley, New York.
13. Craven, P., & Wahba, G. (1979). Smoothing noisy data with spline functions. Numer. Math. 31, 377-390.
14. Cucker, F., & Smale, S. (2002). Best choices for regularization parameters in learning theory: On the bias-variance problem. Foundations of Computational Mathematics, 2(4), 413-428.
15. De Brabanter, J., Pelckmans, K., Suykens, J. A. K., & Vandewalle, J. (2000). Robust cross-validation score function for non-linear function estimation. In Proceedings of the International Conference on Artificial Neural Networks (ICANN 2002), Madrid, Spain, 713-719.
16. De Brabanter, J., Pelckmans, K., Suykens, J. A. K., De Moor, B., & Vandewalle, J. (2003). Robust complexity criteria for nonlinear regression in NARX models. In Proceedings of the 13th System Identification Symposium (SYSID2003), Rotterdam, the Netherlands (pp. 79-84).
17. Golub, G. H., Heath, M., & Wahba, G. (1979). Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics, 21, 215-223.
18. Golub, G. H., & Van Loan, C. F. (1989). Matrix Computations. The John Hopkins University Press.
19. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristics. Radiology, 143, 29-36.
20. Hansen, P. C., (1992). Analysis of discrete ill-posed problems by means of the L-curve. SIAM Review, 34(4), 561-580.
21. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The Elements of Statistical Learning. Springer-Verlag, Heidelberg.
22. Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 12(1), 5582.
23. Ivanov, V. V. (1976). The Theory of Approximate Methods and Their Application to the Numerical Solution of Singular Integral Equations. Nordhoff International.
24. Kearns, M. (1997). A bound on the error of cross validation using the approximation and estimation rates, with consequences for the training-test split. Neural Computation, 9(5), 1143-1161.
25. Lanckriet, G. R. G., Cristianini, N., Bartlett, P., El Ghaoui, L., Jordan, M. I. (2004). Learning the Kernel Matrix with semidefinite programming, Journal of Machine Learning Research, 5(Jan), 27-72.
26. MacKay, D. J. C. (1992). The evidence framework applied to classification networks. Neural Computation, 4, 698-714.
27. MacKay, D. J. C. (1998). Introduction to Gaussian processes. In Neural networks and machine learning (Ed. C.M. Bishop), Springer NATO-ASI Series F: Computer and Systems Sciences, 168, 133-165.
28. p. Technometrics, 15, 661-675.
29. Morozov, V. A. (1984). Methods for Solving Incorrectly Posed Problems. Springer-Verlag.
30. Pelckmans, K., De Brabanter, J., Suykens, J. A. K., & De Moor, B. (2003). Variogram based noise variance estimation and its use in Kernel Based Regression. In Proceedings of the IEEE Workshop on Neural Networks for Signal Processing, 8, Toulouse, France (pp. 199-208).
31. Perrone, M. P. & Cooper, L. N. (1993). When networks disagree: Ensemble method for neural networks. In R. J. Mammone (Ed.), Neural Networks for Speech and Image Processing, Chapman-Hall.
32. Poggio, T., & Girosi, F. (1990). Networks for approximation and learning. Proceedings of the IEEE, 78, 1481-1497.
33. Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge University Press, Oxford.
34. Rissanen, J. (1978). Modelling by shortest Data Description. Automatica, 14, 465-471.
35. Rockafeller, T. R. (1970). Convex Analysis. Princeton University Press.
36. Saunders, C., Gammerman, A., & Vovk, V. (1998). Ridge regression learning algorithm in dual variables. Proceedings of the 15th Int. Conf. on Machine learning (ICML'98), Morgan Kaufmann, 515-521.
37. Sen, A., & Srivastava, M. (1990). Regression Analysis, Theory, Methods, and Applications. Springer.
38. Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. J. Royal Statist. Soc. Ser. B, 36, 111-147.
39. Suykens, J. A. K., & Vandewalle, J. (1999). Least squares support vector machine classifiers. Neural Processing Letters, 9, 293-300.
40. Suykens, J. A. K., De Brabanter, J., Lukas, L., & Vandewalle, J. (2002a). Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing, Special issue on fundamental and information processing aspects of neurocomputing, 48(1-4), 85-105.
41. Suykens, J. A. K., Van Gestel, T., De Brabanter, J., De Moor, B., & Vandewalle, J. (2002b). Least Squares Support Vector Machines. World Scientific, Singapore.
42. Suykens, J. A. K., Horvath, G., Basu, S., Micchelli, C., Vandewalle, J. (Eds.) (2003) Advances in Learning Theory: Methods, Models and Applications, NATO Science Series III: Computer & Systems Sciences, 190, IDS Press Amsterdam.
43. Schölkopf, B., & Smola, A. (2002). Learning with Kernels. MIT Press, Cambridge, MA.
44. Schumaker, L. L. (1981). Spline Functions: Basic Theory. New York: Wiley.
45. Tikhonov, A., & Arsenin, V. Y. (1977). Solutions of Ill-posed problems. V.H.Winston & Sons, Washington D.C.
46. Van Gestel T. (2002). From Linear to Kernel Based Methods in Classification, Modelling and Prediction. Ph.D. Thesis, Faculty of Engineering, K.U.Leuven (Leuven, Belgium).
47. Vapnik, V. N. (1998). Statistical Learning Theory. John Wiley & Sons.
48. Wahba, G. (1990). Splines Models for Observational Data. Series in Applied Mathematics, 59, SIAM, Philadelphia.