A refined margin analysis for boosting algorithms via equilibrium margin

Journal of Machine Learning Research

Volumn 12, Issue , 2011, Pages 1835-1863

  Wang, Liwei ^a   Sugiyama, Masashi ^b   Jing, Zhaoxiang ^a   Yang, Cheng ^c   Zhou, Zhi Hua ^d   Feng, Jufu ^a  

^a Peking University   (China)

^b TOKYO INSTITUTE OF TECHNOLOGY   (Japan)

^c BEIJING AEROSPACE CONTROL CENTER   (China)

^d NANJING UNIVERSITY   (China)

Author keywords

Boosting; Margin bounds; Voting classifier

Indexed keywords

BENCHMARK DATA; BOOSTING; BOOSTING ALGORITHM; DISTRIBUTION BOUNDS; EMPIRICAL ERRORS; GENERALIZATION ERROR; MARGIN ANALYSIS; MARGIN BOUNDS; MARGIN THEORY; REFINED ANALYSIS; TEST ERRORS; THEORETICAL EXPLANATION; TRAINING DATA; UPPER BOUND; VOTING CLASSIFIERS;

ALGORITHMS; REFINING;

ADAPTIVE BOOSTING;

EID: 79960142790     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (34)

1. A. Asuncion and D. J. Newman. UCI machine learning repository, 2007. URL http://www.ics.uci.edu/-mlearn/MLRepository.html.
2. P. Bartlett, M. Jordan, and J.D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101:138-156, 2006. (Pubitemid 43500031)
3. E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting and variants. Machine Learning, 36:105-139, 1999.
4. L. Breiman. Arcing classifiers. The Annals of Statistics, 26:801-849, 1998. (Pubitemid 128450035)
5. L. Breiman. Prediction games and arcing algorithms. Neural Computation, 11:1493-1517, 1999.
6. L. Breiman. Population theory for boosting ensembles. Annals of Statistics, 32:1-11, 2004.
7. R. Caruana and A. Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In 23th International Conference on Machine Learning, 2006.
8. L. Devroye. Bounds for the uniform deviation of empirical measures. Journal of Multivariate Analysis, 12:72-79, 1982.
9. T. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting and randomization. Machine Learning, 40:139-157, 2000.
10. Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In International Conference on Machine Learning, 1996.
11. Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55:119-139, 1997. (Pubitemid 127433398)
12. J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression: A statistical view of boosting. Annals of Statistics, 28:337-407, 2000. (Pubitemid 33227445)
13. A. J. Grove and D. Schuurmans. Boosting in the limit: Maximizing the margin of learned ensembles. In National Conference on Artificial Intelligence, 1998.
14. W. Hoeffding. Probability inequalities for sum of bounded random variables. Journal of American Statistical Society, 58:13-30, 1963.
15. W. Jiang. Process consistency for adaboost. The Annals of Statistics, 32:13-29, 2004. (Pubitemid 41449303)
16. V. Koltchinskii and D. Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of Statistics, 30:1-50, 2002. (Pubitemid 37095367)
17. V. Koltchinskii and D. Panchenko. Complexities of convex combinations and bounding the generalization error in classification. Annals of Statistics, 33:1455-1496, 2005. (Pubitemid 41423978)
18. J. Langford. Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 6:273-306, 2005.
19. G. Lugosi and Nicolas Vayatis. On the bayes-risk consistency of regularized boosting methods. The Annals of Statistics, 32:30-55, 2004. (Pubitemid 41449304)
20. D. Mease and A. Wyner. Evidence contrary to the statistical view of boosting. Journal of Machine Learning Research, 9:131-156, 2008. (Pubitemid 351469030)
21. R. Meir and G. Rätsch. An introduction to boosting and leveraging. In Advanced Lectures on Machine Learning, pages 118-183, 2003.
22. J. R. Quinlan. Bagging, boosting, and c4.5. In 13th International Conference on Artificial Intelligence, 1996.
23. G. Rätsch and M. Warmuth. Efficient margin maximization with boosting. Journal of Machine Learning Research, 6:2131-2152, 2005.
24. L. Reyzin and R. E. Schapire. How boosting the margin can also boost classifier complexity. In International Conference on Machine Learning, 2006.
25. C. Rudin, I. Daubechies, and R. Schapire. The dynamics of AdaBoost: Cyclic behavior and convergence of margins. Journal of Machine Learning Research, 5:1557-1595, Dec 2004.
26. C. Rudin, I. Daubechies, and R. Schapire. Analysis of boosting algorithms using the smooth margin function. Annals of Statistics, 35:2723-2768, 2007.
27. N. Sauer. On the density of family of sets. Journal of Combinatorial Theory, Series A, 13:145-147, 1972.
28. R. Schapire, Y. Freund, P. Bartlett, and W. Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. Annals of Statistics, 26:1651-1686, 1998. (Pubitemid 128376902)
29. V. Vapnik. Statistical Learning Theory. John Wiley and Sons Inc., 1998.
30. V. N. Vapnik and A. YA. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16:264-280, 1971.
31. P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001.
32. L. Wang, C. Yang, and J. Feng. On learning with dissimilarity functions. In 24th International Conference on Machine Learning, 2007.
33. L. Wang, M. Sugiyama, C. Yang, Z. Zhou, and J. Feng. On the margin explanation of boosting algorithms. In 21th Annual Conference on Learning Theory, 2008.
34. T. Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 32:56-85, 2004.