Complexities of convex combinations and bounding the generalization error in classification

Annals of Statistics

Volumn 33, Issue 4, 2005, Pages 1455-1496

  Koltchinskii, Vladimir ^a   Panchenko, Dmitry ^b  

^a UNIVERSITY OF NEW MEXICO   (United States)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Classification; Convex combination; Convex hull; Empirical margin distribution; Empirical process; Generalization error; Margin

Indexed keywords

EID: 26444607491     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/009053605000000228     Document Type: Article

References (34)

1. ANDONOVA, S. (2004). Theoretical and experimental analysis of the generalization ability of some statistical learning algorithms. Ph.D. dissertation, Boston Univ.
2. ANTOS, A., KÉGL, B., LINDER, T. and LUGOSI, G. (2002). Data-dependent margin-based generalization bounds for classification. J. Mach. Learn. Res. 3 73-98.
3. AUDIBERT, J.-Y. (2004). Aggregated estimators and empirical complexity for least square regression. Ann. Inst. H. Poincaré Probab. Statist. 40 685-736.
4. BARTLETT, P. (1998). The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. Inform. Theory 44 525-536.
5. BARTLETT, P., BOUSQUET, O. and MENDELSON, S. (2005). Local Rademacher complexities. Ann. Statist. 33 1497-1537.
6. BARTLETT, P., JORDAN, M. and MCAULIFFE, J. (2005). Convexity, classification and risk bounds. J. Amer. Statist. Assoc. To appear.
7. BLANCHARD, G., LUGOSI, G. and VAYATIS, N. (2004). On the rate of convergence of regularized boosting classifiers. J. Mach. Learn. Res. 4 861-894.
8. BOUSQUET, O., KOLTCHINSKII, V. and PANCHENKO, D. (2002). Some local measures of complexity of convex hulls and generalization bounds. Computational Learning Theory. Lecture Notes in Artificial Intelligence 2375 59-73. Springer, Berlin.
9. BREIMAN, L. (1996). Bagging predictors. Machine Learning 24 123-140.
10. BREIMAN, L. (1998). Arcing classifiers (with discussion). Ann. Statist. 26 801-849.
11. BREIMAN, L. (1999). Prediction games and arcing algorithms. Neural Computation 11 1493-1517.
12. DEVROYE, L., GYÖRFI, L. and LUGOSI, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer, New York.
13. DUDLEY, R. M. (1999). Uniform Central Limit Theorems. Cambridge Univ. Press.
14. HAUSSLER, D. (1995). Sphere packing numbers for subsets of the Boolean n-cube with bounded Vapnik-Chervonenkis dimension. J. Combin. Theory Ser. A 69 217-232.
15. JIANG, W. (2004). Process consistency for AdaBoost. Ann. Statist. 32 13-29.
16. KOLTCHINSKII, V. (2003). Bounds on margin distributions in learning problems. Ann. Inst. H. Poincaré Probab. Statist. 39 943-978.
17. KOLTCHINSKII, V. (2003). Local Rademacher complexities and oracle inequalities in risk minimization. Preprint.
18. KOLTCHINSKII, V. and PANCHENKO, D. (2000). Rademacher processes and bounding the risk of function learning. In High Dimensional Probability II (E. Giné, D. Mason and J. Wellner, eds.) 443-459. Birkhäuser, Boston.
19. KOLTCHINSKII, V. and PANCHENKO, D. (2002). Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Statist. 30 1-50.
20. KOLTCHINSKII, V., PANCHENKO, D. and ANDONOVA, S. (2003). Generalization bounds for voting classifiers based on sparsity and clustering. Lecture Notes in Artificial Intelligence. To appear.
21. KOLTCHINSKII, V., PANCHENKO, D. and LOZANO, F. (2003). Bounding the generalization error of convex combinations of classifiers: Balancing the dimensionality and the margins. Ann. Appl. Probab. 13 213-252.
22. LUGOSI, G. and VAYATIS, N. (2004). On the Bayes-risk consistency of regularized boosting methods. Ann. Statist. 32 30-55.
23. MASSART, P. (2000). Some applications of concentration inequalities to statistics. Probability theory. Ann. Fac. Sci. Toulouse Math. (6) 9 245-303.
24. PANCHENKO, D. (2001). A note on Talagrand's concentration inequality. Electron. Comm. Probab. 6 55-65.
25. PANCHENKO, D. (2002). Some extensions of an inequality of Vapnik and Chervonenkis. Electron. Comm. Probab. 7 55-65.
26. PANCHENKO, D. (2003). Symmetrization approach to concentration inequalities for empirical processes. Ann. Probab. 31 2068-2081.
27. PISIER, G. (1981). Remarques sur un résultat non publié de B. Maurey. In Seminar on Functional Analysis, 1980-1981, École Polytechnic, Palaiseau. Exp. no. V, 13 pp.
28. SCHAPIRE, R., FREUND, Y., BARTLETT, P. and LEE, W. S. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26 1651-1686.
29. SCHAPIRE, R. and SINGER, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning 37 297-336.
30. STEINWART, I. (2004). Sparseness of support vector machines. J. Mach. Learn. Res. 4 1071-1105.
31. TSYBAKOV, A. (2003). Optimal rates of aggregation. Lecture Notes in Artificial Intelligence. To appear.
32. VAN DER VAART, A. W. and WELLNER, J. A. (1996). Weak Convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
33. ZHANG, T. (2004). Statistical behavior and consistency of classification methods based on convex risk minimization. Ann. Statist. 32 56-85.
34. ZHANG, T. and YU, B. (2005). Boosting with early stopping: Convergence and consistency. Ann. Statist. 33 1538-1579.