Empirical margin distributions and bounding the generalization error of combined classifiers

Annals of Statistics

Volumn 30, Issue 1, 2002, Pages 1-50

  Koltchinskii, V ^a   Panchenko, D ^a  

^a UNIVERSITY OF NEW MEXICO   (United States)

Author keywords

Boosting; Combined classifier; Concentration inequalities; Empirical process; Gaussian process; Generalization error; Margin; Neural network; Rademacher process

Indexed keywords

EID: 0036104545     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (37)

1. ANTHONY, M. and BARTLETT, P. (1999). Neural Network Learning: Theoretical Foundations. Cambridge Univ. Press.
2. BARRON, A. (1991a). Complexity regularization with applications to artificial neural networks. In Nonparametric Functional Estimation and Related Topics (G. Roussas, ed.) 561-576. Kluwer, Dordrecht.
3. BARRON, A. (1991b). Approximation and estimation bounds for artificial neural networks. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory 243-249. Morgan Kaufmann, San Francisco.
4. BARRON, A., BIRGÉ, L. and MASSART, P. (1999). Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 301-413.
5. BARTLETT, P. (1998). The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans. Inform. Theory 44 525-536.
6. BARTLETT, P. and SHAWE-TAYLOR, J. (1999). Generalization performance of support vector machines and other pattern classifiers. In Advances in Kernel Methods: Support Vector Learning (B. Schölkopf, C. Burges and A. J. Smola, eds.) 43-54. MIT Press.
7. BIRGÉ, L. and MASSART, P. (1997). From model selection to adaptive estimation. In Festschrift for Lucien Le Cam: Research Papers in Probability and Statistics (D. Pollard, E. Torgersen and G. Yang, eds.) 55-87. Springer, New York.
8. BLAKE, C. and MERZ, C. (1998). UCI repository of machine learning databases. Available at www.ics.uci.edu/∼mlearn/MLRepository.html.
9. BREIMAN, L. (1996). Bagging predictors. Machine Learning 24 123-140.
10. CORTES, C. and VAPNIK, V. (1995). Support vector networks. Machine Learning 20 273-297.
11. DEVROYE, L., GYÖRFI, L. and LUGOSI, G. (1996). A Probabilistic Theory of Pattern Recognition. Springer, New York.
12. DUDLEY, R. M. (1999). Uniform Central Limit Theorems. Cambridge Univ. Press.
13. FELLER, W. (1950). An Introduction to Probability Theory and Its Applications 1. Wiley, New York.
14. FINE, T. L. (1999). Feedforward Neural Network Methodology. Springer, New York.
15. FREUND, Y. (1995). Boosting a weak learning algorithm by majority. Inform, and Comput. 121 256-285.
16. FREUND, Y. (1999). An adaptive version of the boost by majority algorithm. Preprint.
17. FRIEDMAN, J. (1999). Greedy function approximation: a gradient boosting machine. Technical report, Dept. Statistics, Stanford Univ.
18. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2000). Additive logistic regression: a statistical view of boosting. Ann. Statist. 28 337-374.
19. JOHNSTONE, I. M. (1998). Oracle inequalities and nonparametric function estimation. In Doc. Math. (Extra Volume) 3 267-278.
20. KOLTCHINSKII, V. (2001). Rademacher penalties and structural risk minimization. IEEE Trans. Inform. Theory. To appear.
21. KOLTCHINSKII, V. and PANCHENKO, D. (2000). Rademacher processes and bounding the risk of function learning. In High Dimensional Probability II (E. Giné, D. Mason and J. Wellner, eds.) 444-459. Birkhäuser, Boston.
22. KOLTCHINSKII, V., PANCHENKO, D. and LOZANO, F. (2000a). Bounding the generalization error of convex combinations of classifiers: balancing the dimensionality and the margins. Preprint. Available at www.boosting.org/.
23. KOLTCHINSKII, V., PANCHENKO, D. and LOZANO, F. (2000b). Some new bounds on the generalization error of combined classifiers. Advances in Neural Information Processing Systems. Available at www.boosting.org/.
24. LEDOUX, M. and TALAGRAND, M. (1991). Probability in Banach Spaces. Springer, New York.
25. MASON, L., BARTLETT, P. and BAXTER, J. (1999). Improved generalization through explicit optimization of margins. Machine Learning 38 243-255.
26. MASON, L., BAXTER, J., BARTLETT, P. and FREAN, M. (2000). Functional gradient techniques for combining hypotheses. In Advances in Large Margin Classifiers (A. J. Smola, P. Bartlett, B. Schölkopf and D. Schurmans, eds.) MIT Press.
27. MASSART, P. (2000). About the constants in Talagrand's concentration inequalities for empirical processes. Ann. Probab. 28 863-885.
28. SAZONOV, V. V. (1963). On the Glivenko-Cantelli theorem. Theory Probab. Appl. 8 282-285.
29. SCHAPIRE, R., FREUND, Y., BARTLETT, P. and LEE, W. S. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26 1651-1686.
30. TALAGRAND, M. (1996a). A new look at independence. Ann. Probab. 24 1-34.
31. TALAGRAND, M. (1996b). New concentration inequalities in product spaces. Invent. Math. 126 505-563.
32. TALAGRAND, M. (1998). Rigorous results for the Hopfield model with many patterns. Probab. Theory Related Fields 110 177-276.
33. TOPSØE, F., DUDLEY, R. and HOFFMANN-JØRGENSEN, J. (1976). Two examples concerning uniform convergence of measures w.r.t. balls in Banach spaces. Empirical Distributions and Processes. Lecture Notes in Math. 566 141-146. Springer, Berlin.
34. VAN DER VAART, A. W. and WELLNER, J. A. (1996). Weak convergence and Empirical Processes. With Applications to Statistics. Springer, New York.
35. VAPNIK, V. (1998). Statistical Learning Theory. Wiley, New York.
36. VIDYASAGAR, M. (1997). A Theory of Learning and Generalization. Springer, New York.
37. YUKICH, J., STINCHCOMBE, H. and WHITE, H. (1995). Sup-norm approximation bounds for networks through probabilistic methods. IEEE Trans. Inform. Theory 41 1021-1027.