On the rate of convergence of regularized boosting classifiers

Journal of Machine Learning Research

Volumn 4, Issue 5, 2004, Pages 861-894

  Blanchard, Gilles ^a   Lugosi, Gábor ^b   Vayatis, Nicolas ^c  

^a UNIVERSITÉ PARIS SACLAY   (France)

^b UNIVERSITAT POMPEU FABRA   (Spain)

^c SORBONNE UNIVERSITÉ   (France)

Author keywords

Boosting; Classification; Consistency; Decision stumps; Rates of convergence

Indexed keywords

ERROR ANALYSIS; FUNCTIONS; MATHEMATICAL MODELS; PROBABILITY; PROBLEM SOLVING; THEOREM PROVING;

BOOSTING; CLASSIFICATION; CONSISTENCY; DECISION STUMPS; RATES OF CONVERGENCE;

CONVERGENCE OF NUMERICAL METHODS;

EID: 3042675892     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/1532443041424319     Document Type: Conference Paper

References (48)

1. A.R. Barron. Neural net approximation. In Yale Workshop on Adaptive and Learning Systems, 1992.
2. A.R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39:930-944, 1993.
3. P.L. Bartlett, O. Bousquet and S. Mendelson. Localized Rademacher complexities. Proceedings of the 15th annual conference on Computational Learning Theory, 44-58, 2002.
4. P.L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Manuscript, 2003.
5. P.L. Bartlett and S. Mendelson. Rademacher and Gaussian complexities: risk bounds and structural results. Journal of Machine Learning Research 3, 463-482, 2002.
6. K. Bennett, A. Demiriz, and G. Rátsch. Sparse regression ensembles in infinite and finite hypothesis spaces. Machine Learning, 48:193-221, 2002.
7. L. Birgé and P. Massart. Minimum contrast estimators on sieves: exponential bounds and rates of convergence. Bernoulli 4, 329-375, 1998.
8. O. Bousquet. New Approaches to Statistical Learning Theory. Annals of the Institute of Statistical Mathematics, 2003, to appear.
9. G. Blanchard, O. Bousquet, and P. Massart. Statistical performance of Support Vector Machines. Manuscript, 2003.
10. L. Breiman. Arcing classifiers. Annals of Statistics, 26:801-849, 1998.
11. L. Breiman. Some infinite theory for predictor ensembles. Technical Report 577, Statistics Department, UC Berkeley, 2000.
12. 2-loss: regression and classification. J. Amer. Statist. Assoc., 2003, to appear.
13. M. Collins, R.E. Schapire, and Y. Singer. Logistic regression, AdaBoost and Bregman distances. In Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, 2000.
14. G. Cybenko. Approximations by superpositions of sigmoidal functions. Math. Control, Signals, Systems, 2:303-314, 1989.
15. C. Darken, M. Donahue, L. Gurvits, and E. Sontag. Rates of convex approximation in non-Hilbert spaces. Constructive Approximation, 13(2):187-220, 1997.
16. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer-Verlag, New York, 1996.
17. R.M. Dudley. Central limit theorems for empirical measures. Annals of Probability, 6:899-929, 1978.
18. N. Duffy and D. Helmbold. Potential boosters? In S.A. Solla, T.K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems 12, pages 258-264. MIT Press, 2000.
19. Y. Freund. Boosting a weak learning algorithm by majority. Information and Computation, 121:256-285, 1995.
20. Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55:119-139, 1997.
21. J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: a statistical view of boosting. Annals of Statistics, 28:337-374, 2000.
22. F. Girosi and G. Anzelloti. Rates of convergence for radial basis functions and neural networks. Artificial Neural Networks for Speech and Vision, p.169-176, Chapman and Hall, 1993.
23. T. Hastie and R.J. Tibshirani. Generalized Additive Models. Chapman and Hall, London, U. K., 1990.
24. K. Hornik, M. Stinchcombe, and H. White. Multi-layer feedforward networks are universal approximators. Neural Networks, 2:359-366, 1989.
25. W. Jiang. Process consistency for AdaBoost. Annals of Statistics, 2003, to appear (with discussion).
26. V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of Statistics, 30, 2002.
27. M. Ledoux and M. Talagrand. Probability in Banach Spaces. Springer-Verlag, New York, 1991.
28. G. Lugosi and S. Mendelson. A note on the richness of convex hulls of VC classes. Manuscript, 2003.
29. G. Lugosi and N. Vayatis. On the Bayes-risk consistency of regularized boosting methods. Annals of Statistics, 2003, to appear (with discussion).
30. V. Maiorov, R. Meir, and J. Ratsaby. On the approximation of functional classes equipped with a uniform measure using ridge functions. Jour. of Approximation Theory, 99:95-111, 1999.
31. S. Mannor and R. Meir. Weak learners and improved convergence rate in boosting. In Advances in Neural Information Processing Systems 13: Proc. NIPS'2000, 2001.
32. S. Mannor, R. Meir, and T. Zhang. The consistency of greedy algorithms for classification. In Proceedings of the 15th Annual Conference on Computational Learning Theory, 2002.
33. P. Massart. Some applications of concentration inequalities in statistics. Annales de la Faculté des Sciences de Toulouse, Mathématiques, 9(2):245-303, 2000.
34. L. Mason, J. Baxter, P.L. Bartlett, and M. Frean. Functional gradient techniques for combining hypotheses. In A.J. Smola, P.L. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 221-247. MIT Press, Cambridge, MA, 1999.
35. R. Meir and V. Maiorov. On the Optimality of neural network approximation using incremental algorithms. IEEE Trans. Neural Network, 11(2):323-337, 2000.
36. R. Meir and G. Rätsch. An introduction to boosting and leveraging. In S. Mendelson and A. Smola, editors, Advanced Lectures on Machine Learning, LNCS, pages 119-184. Springer, 2003. (In press).
37. S. Mendelson. Improving the sample complexity using global data, IEEE Transactions on Information Theory 48(7), 1977-1991, 2002.
38. E. Nedelec and P. Massart. Risk bounds for statistical learning. Manuscript, 2003.
39. A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8143-196, 1999.
40. D. Pollard. Convergence of Stochastic Processes, Springer-Verlag, New York, 1984.
41. R.E. Schapire. The strength of weak learnability. Machine Learning, 5:197-227, 1990.
42. R.E. Schapire, Y. Freund, P. Bartlett, and W.S. Lee. Boosting the margin: a new explanation for the effectiveness of voting methods. Annals of Statistics, 26:1651-1686, 1998.
43. E. Sontag. Feedback stabilization using two-hidden-layer nets. IEEE Trans. Neural Networks, 3:981-990, 1992.
44. A. B. Tsybakov. Optimal aggregation of classifiers in statistical learning. Annals of Statistics, to appear, 2003.
45. A.W. van der Vaart and J.A. Wellner. Weak convergence and empirical processes. Springer-Verlag, New York, 1996.
46. Y. Yang. Minimax nonparametric classification - part I: rates of convergence. IEEE Transaction on Information Theory, vol. 45, pp. 2271-2284, 1999.
47. Y. Yang. Minimax nonparametric classification - part II: model selection for adaptation. IEEE Transaction on Information Theory, vol. 45, pp. 2285-2292, 1999.
48. T. Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. Annals of Statistics, 2003, to appear (with discussion).