Fully Empirical and Data-Dependent Stability-Based Bounds

IEEE Transactions on Cybernetics

Volumn 45, Issue 9, 2015, Pages 1913-1926

  Oneto, Luca ^a   Ghio, Alessandro ^a   Ridella, Sandro ^a   Anguita, Davide ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

Algorithmic stability; data dependent bounds; fully empirical bounds; in sample; model selection; out of sample; support vector machine (SVM)

Indexed keywords

ALGORITHMS; CLASSIFICATION (OF INFORMATION); LEARNING ALGORITHMS; STABILITY;

ALGORITHMIC STABILITY; DATA DEPENDENT; EMPIRICAL BOUNDS; GENERALIZATION ABILITY; LEARNING PROCEDURES; MODEL SELECTION; STRUCTURAL RISK MINIMIZATION; SVM CLASSIFICATION;

SUPPORT VECTOR MACHINES;

EID: 84939779196     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2361857     Document Type: Article

References (85)

1. O. Bousquet and A. Elisseeff, "Stability and generalization, " J. Mach. Learn. Res., vol. 2, pp. 499-526, Mar. 2002.
2. T. Poggio, R. Rifkin, S. Mukherjee, and P. Niyogi, "General conditions for predictivity in learning theory, " Nature, vol. 428, no. 6981, pp. 419-422, 2004.
3. S. Mukherjee, P. Niyogi, T. Poggio, and R. Rifkin, "Learning theory: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization, " Adv. Comput. Math., vol. 25, no. 1, pp. 161-193, 2006.
4. V. N. Vapnik, Statistical Learning Theory. New York, NJ, USA: Wiley, 1998.
5. R. Herbrich and T. Graepel, "A PAC-Bayesian margin bound for linear classifiers, " IEEE Trans. Inf. Theory, vol. 48, no. 12, pp. 3140-3150, Dec. 2002.
6. D. A. McAllester, "PAC-Bayesian stochastic model selection, " Mach. Learn., vol. 51, no. 1, pp. 5-21, 2003.
7. E. Parrado-Hernández, A. Ambroladze, J. Shawe-Taylor, and S. Sun, "PAC-Bayes bounds with data dependent priors, " J. Mach. Learn. Res., vol. 13, pp. 3507-3531, Dec. 2012.
8. V. N. Vapnik, "An overview of statistical learning theory, " IEEE Trans. Neural Netw., vol. 10, no. 5, pp. 988-999, Sep. 1999.
9. J. Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony, "Structural risk minimization over data-dependent hierarchies, " IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 1926-1940, Jan. 1998.
10. P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results, " J. Mach. Learn. Res., vol. 3, pp. 463-482, Mar. 2003.
11. P. Bartlett, O. Bousquet, and S. Mendelson, "Localized Rademacher complexities, " in Computational Learning Theory. Berlin, Germany: Springer, 2002.
12. D. Anguita, A. Ghio, L. Oneto, and S. Ridella, "In-sample and out-of-sample model selection and error estimation for support vector machines, " IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 9, pp. 1390-1406, Jun. 2012.
13. P. L. Bartlett, S. Boucheron, and G. Lugosi, "Model selection and error estimation, " Mach. Learn., vol. 48, nos. 1-3, pp. 85-113, 2002.
14. I. Guyon, A. Saffari, G. Dror, and G. Cawley, "Model selection: Beyond the Bayesian/frequentist divide, " J. Mach. Learn. Res., vol. 11, pp. 61-87, Jan. 2010.
15. W. H. Rogers and T. J. Wagner, "A finite sample distribution-free performance bound for local discrimination rules, " Ann. Stat., vol. 6, no. 3, pp. 506-514, 1978.
16. J. R. Quinlan, C4. 5: Programs for Machine Learning, vol. 1. San Francisco, CA, USA: Morgan Kaufmann, 1993.
17. H.-W. Hu, Y.-L. Chen, and K. Tang, "A novel decision-tree method for structured continuous-label classification, " IEEE Trans. Cybern., vol. 43, no. 6, pp. 1734-1746, Jan. 2013.
18. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks, " Science, vol. 313, no. 5786, pp. 504-507, 2006.
19. R. Zhang, Y. Lan, G.-B. Huang, Z.-B. Xu, and Y. C. Soh, "Dynamic extreme learning machine and its approximation capability, " IEEE Trans. Cybern., vol. 43, no. 6, pp. 2054-2065, Feb. 2013.
20. P. Klesk and M. Korzen, "Sets of approximating functions with finite Vapnik-Chervonenkis dimension for nearest-neighbors algorithms, " Pattern Recognit. Lett., vol. 32, no. 14, pp. 1882-1893, 2011.
21. B. Efron, The Jackknife, the Bootstrap and Other Resampling Plans, vol. 38. Philadelphia, PA, USA: SIAM, 1982.
22. R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection, " in Proc. Int. Joint Conf. Artif. Intell., San Francisco, CA, USA, 1995, pp. 1137-1143.
23. U. M. Braga-Neto and E. R. Dougherty, "Is cross-validation valid for small-sample microarray classification?" Bioinformatics, vol. 20, no. 3, pp. 374-380, 2004.
24. Q. Wu and D.-X. Zhou, "Learning with sample dependent hypothesis spaces, " Comput. Math. Appl., vol. 56, no. 11, pp. 2896-2907, 2008.
25. A. Cannon, J. M. Ettinger, D. Hush, and C. Scovel, "Machine learning with data dependent hypothesis classes, " J. Mach. Learn. Res., vol. 2, pp. 335-358, Jan. 2002.
26. A. Rakhlin, S. Mukherjee, and T. Poggio, "Stability results in learning theory, " Anal. Appl., vol. 3, no. 4, pp. 397-417, 2005.
27. S. Agarwal and P. Niyogi, "Stability and generalization of bipartite ranking algorithms, " in Proc. 18th Annu. Conf. Learn. Theory, Berlin, Germany, 2005, pp. 32-47.
28. M. Kearns and D. Ron, "Algorithmic stability and sanity-check bounds for leave-one-out cross-validation, " Neural Comput., vol. 11, no. 6, pp. 1427-1453, 1999.
29. A. Elisseeff, T. Evgeniou, and M. Pontil, "Stability of randomized learning algorithms, " J. Mach. Learn. Res., vol. 6, no. 1, p. 55, 2006.
30. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, "Learnability, stability and uniform convergence, " J. Mach. Learn. Res., vol. 11, pp. 2635-2670, Mar. 2010.
31. C. Cortes and V. Vapnik, "Support-vector networks, " Mach. Learn., vol. 20, no. 3, pp. 273-297, 1995.
32. L. Györfi, A Distribution-Free Theory of Nonparametric Regression. London, U.K.: Springer, 2002.
33. R. Adamczak, "A tail inequality for suprema of unbounded empirical processes with applications to Markov chains, " Electron. J. Probab., vol. 13, no. 34, pp. 1000-1034, 2007.
34. S. Kutin, "Extensions to McDiarmid's inequality when differences are bounded with high probability, " Dept. Comp. Sci., Univ. Chicago, Chicago, IL, USA, Tech. Rep. TR-2002-04, 2002.
35. K. Fukunaga and D. M. Hummels, "Leave-one-out procedures for non-parametric error estimates, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 4, pp. 421-423, Aug. 1989.
36. M. M. Lee, S. S. Keerthi, C. J. Ong, and D. DeCoste, "An efficient method for computing leave-one-out error in support vector machines with Gaussian kernels, " IEEE Trans. Neural Netw., vol. 15, no. 3, pp. 750-757, May 2004.
37. C. McDiarmid, "On the method of bounded differences, " Surv. Comb., vol. 141, no. 1, pp. 148-188, 1989.
38. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition, vol. 31. New York, NY, USA: Springer, 1996.
39. L. Devroye and T. Wagner, "Distribution-free inequalities for the deleted and holdout error estimates, " IEEE Trans. Inf. Theory, vol. 25, no. 2, pp. 202-207, Jan. 1979.
40. H. Drucker, C. J. Burges, L. Kaufman, A. Smola, and V. Vapnik, "Support vector regression machines, " Advances in Neural Information Processing Systems. San Mateo, CA, USA: Morgan Kaufmann, 1997, pp. 155-161.
41. I. Steinwart, "Consistency of support vector machines and other regularized kernel classifiers, " IEEE Trans. Inf. Theory, vol. 51, no. 1, pp. 128-142, Jan. 2005.
42. R. Dietrich, M. Opper, and H. Sompolinsky, "Statistical mechanics of support vector networks, " Phys. Rev. Lett., vol. 82, no. 14, p. 2975, 1999.
43. M. Opper, W. Kinzel, J. Kleinz, and R. Nehl, "On the ability of the optimal perceptron to generalise, " J. Phys. A. Math. Gen., vol. 23, no. 11, p. L581, 1990.
44. M. Opper, "Statistical mechanics of learning: Generalization, " in The Handbook of Brain Theory and Neural Networks, 2nd ed, Cambridge, MA, USA: MIT Press, 2002, pp. 922-925.
45. S. Mukherjee et al., "Estimating dataset size requirements for classifying DNA microarray data, " J. Comput. Biol., vol. 10, no. 2, pp. 119-142, 2003.
46. W. Hoeffding, "Probability inequalities for sums of bounded random variables, " J. Amer. Stat. Assoc., vol. 58, no. 301, pp. 13-30, 1963.
47. M. Aupetit, "Nearly homogeneous multi-partitioning with a deterministic generator, " Neurocomputing, vol. 72, nos. 7-9, pp. 1379-1389, 2009.
48. D. Anguita, L. Ghelardoni, A. Ghio, L. Oneto, and S. Ridella, "The 'K' in K-fold cross validation, " in Proc. Eur. Symp. Artif. Neural Netw., Bruges, Belgium, 2011, pp. 441-446.
49. D. Anguita, S. Ridella, and F. Rivieccio, "K-fold generalization capability assessment for support vector classifiers, " in Proc. IEEE Int. Joint Conf. Neural Netw., Montreal, QC, Canada, 2005, pp. 855-858.
50. B. Milenova, J. Yarmus, and M. Campos, "SVM in Oracle database 10g: Removing the barriers to widespread adoption of support vector machines, " in Proc. 31st Int. Conf. Very Large Data Bases, Trondheim, Norway, 2005, pp. 1152-1163.
51. Z. Xu, M. Dai, and D. Meng, "Fast and efficient strategies for model selection of Gaussian support vector machine, " IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 39, no. 5, pp. 1292-1307, Mar. 2009.
52. T. Glasmachers and C. Igel, "Maximum likelihood model selection for 1-norm soft margin SVMS with multiple parameters, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 8, pp. 1522-1528, Apr. 2010.
53. K. De Brabanter, J. De Brabanter, J. Suykens, and B. De Moor, "Approximate confidence and prediction intervals for least squares support vector regression, " IEEE Trans. Neural Netw., vol. 22, no. 1, pp. 110-120, Nov. 2011.
54. M. Karasuyama and I. Takeuchi, "Nonlinear regularization path for quadratic loss support vector machines, " IEEE Trans. Neural Netw., vol. 22, no. 10, pp. 1613-1625, Aug. 2011.
55. J. Platt, "Fast training of support vector machines using sequential minimal optimization, " in Advances in Kernel Methods, B. Schölkopf, C. J. C. Burges, and A. J. Smola, Eds. Cambridge, MA, USA: MIT Press, 1999, pp. 185-208.
56. L. J. Cao et al., "Parallel sequential minimal optimization for the training of support vector machines, " IEEE Trans. Neural Netw., vol. 17, no. 4, pp. 1039-1049, Jul. 2006.
57. B. Scholkopf, "The kernel trick for distances, " in Proc. Adv. Neural Inf. Process. Syst., 2001, pp. 1299-1319.
58. S. Keerthi and C. Lin, "Asymptotic behaviors of support vector machines with Gaussian kernel, " Neural Comput., vol. 15, no. 7, pp. 1667-1689, 2003.
59. S. Arlot and A. Celisse, "A survey of cross-validation procedures for model selection, " Stat. Surv., vol. 4, pp. 40-79, 2010.
60. J. D. Rodriguez, A. Perez, and J. A. Lozano, "Sensitivity analysis of k-fold cross validation in prediction error estimation, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 3, pp. 569-575, Mar. 2010.
61. J. G. Moreno-Torres, J. A. Sáez, and F. Herrera, "Study on the impact of partition-induced dataset shift on k-fold cross-validation, " IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 8, pp. 1304-1312, Jun. 2012.
62. D. Anguita, A. Ghio, S. Ridella, and D. Sterpi, "K-fold cross validation for error rate estimate in support vector machines, " in Proc. Int. Conf. Data Mining, Las Vegas, NV, USA, 2009, pp. 291-297.
63. J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge, U.K.: Cambridge Univ. Press, 2004.
64. I. Steinwart and A. Christmann, Support Vector Machines. Berlin, Germany: Springer, 2008.
65. H. Akaike, "A new look at the statistical model identification, " IEEE Trans. Autom. Control, vol. 19, no. 6, pp. 716-723, Dec. 1974.
66. T. Joachims, "Training linear SVMS in linear time, " in Proc. Int. Conf. Knowl. Disc. Data Mining, Philadelphia, PA, USA, 2006, pp. 217-226.
67. R. Collobert, S. Bengio, and Y. Bengio, "A parallel mixture of SVMS for very large scale problems, " Neural Comput., vol. 14, no. 5, pp. 1105-1114, 2002.
68. I. W. Tsang, J. T. Kwok, and P.-M. Cheung, "Very large SVM training using core vector machines, " in Proc. Int. Workshop Artif. Intell. Stat., 2005, pp. 349-356.
69. J. Shawe-Taylor and S. Sun, "A review of optimization methodologies in support vector machines, " Neurocomputing, vol. 74, no. 17, pp. 3609-3618, 2011.
70. S. S. Keerthi, S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy, "Improvements to Platt's SMO algorithm for SVM classifier design, " Neural Comput., vol. 13, no. 3, pp. 637-649, 2001.
71. R.-E. Fan, P.-H. Chen, and C.-J. Lin, "Working set selection using second order information for training support vector machines, " J. Mach. Learn. Res., vol. 6, pp. 1889-1918, Dec. 2005.
72. G. Cauwenberghs and T. Poggio, "Incremental and decremental support vector machine learning, " in Proc. Adv. Neural Inf. Process. Syst., 2001, pp. 409-415.
73. M. Karasuyama and I. Takeuchi, "Multiple incremental decremental learning of support vector machines, " in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 1048-1059.
74. J. A. Blackard and D. J. Dean, "Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables, " Comput. Electron. Agr., vol. 24, no. 3, pp. 131-151, 1999.
75. S. Munder and D. M. Gavrila, "An experimental study on pedestrian classification, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1863-1868, Nov. 2006.
76. L. Bottou et al., "Comparison of classifier methods: A case study in handwritten digit recognition, " in Proc. Int. Conf. Pattern Recognit. Conf. B Comput. Vis. Image Process., Jerusalem, Israel, Oct. 1994, pp. 77-82.
77. Y. Bulatov, (2011). "Notmnist dataset, " Google (Books/OCR), Tech. Rep. [Online]. Available: http://yaroslavvb.blogspot.it/2011/09/notmnist-dataset.html
78. C.-W. Hsu and C.-J. Lin, "A comparison of methods for multiclass support vector machines, " IEEE Trans. Neural Netw., vol. 13, no. 2, pp. 415-425, Mar. 2002.
79. C.-W. Hsu, C.-C. Chang, and C.-J. Lin, "A practical guide to support vector classification, " Dept. Comput. Sci., Nat. Taiwan Univ., Tech. Rep., 2003 [Online]. Available http://www.csie.ntu.edu.tw/∼cjlin/papers/guide/guide.pdf
80. M. Talagrand, "A new look at independence, " Ann. Probab., vol. 24, no. 1, pp. 1-34, 1996.
81. S. Boucheron, G. Lugosi, and P. Massart, "A sharp concentration inequality with applications, " Random Struct. Algorithms, vol. 16, no. 3, pp. 277-292, May 2000.
82. O. Bousquet, "A Bennett concentration inequality and its application to suprema of empirical processes, " CR Math., vol. 334, no. 6, pp. 495-500, 2002.
83. N. Srebro, K. Sridharan, and A. Tewari, "Smoothness, low-noise, and fast rates, " in Proc. Neural Inf. Process. Syst., 2010, pp. 2199-2207.
84. E. Mammen and A. B. Tsybakov, "Smooth discrimination analysis, " Ann. Stat., vol. 27, no. 6, pp. 1808-1829, 1999.
85. J.-Y. Audibert, "Fast learning rates in statistical inference through aggregation, " Ann. Stat., vol. 37, no. 4, pp. 1591-1646, 2009.