On the doubt about margin explanation of boosting

Artificial Intelligence

Volumn 203, Issue , 2013, Pages 1-18

  Gao, Wei ^a   Zhou, Zhi Hua ^a  

^a NANJING UNIVERSITY   (China)

Author keywords

Boosting; Classification; Ensemble methods; Margin theory

Indexed keywords

BOOSTING; ENSEMBLE METHODS; GENERALIZATION ERROR; GENERALIZATION ERROR BOUNDS; MARGIN DISTRIBUTIONS; MARGIN THEORY; TRAINING ERRORS; VOTING CLASSIFIERS;

ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION);

ADAPTIVE BOOSTING;

EID: 84881447722     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.artint.2013.07.002     Document Type: Article

References (47)

1. A. Antos, B. Kégl, T. Linder, and G. Lugosi Data-dependent margin-based generalization bounds for classification Journal of Machine Learning Research 3 2002 73 98
2. J.Y. Audibert, R. Munos, and C. Szepesvári Exploration- exploitation tradeoff using variance estimates in multi-armed bandits Theoretical Computer Science 410 19 2009 1876 1902
3. P.L. Bartlett, M.I. Jordan, and J.D. McAuliffe Convexity, classification, and risk bounds Journal of the American Statistical Association 101 473 2006 138 156 (Pubitemid 43500031)
4. P.L. Bartlett, and M. Traskin Adaboost is consistent Journal of Machine Learning Research 8 2007 2347 2368 (Pubitemid 350046195)
5. E. Bauer, and R. Kohavi An empirical comparison of voting classification algorithms: Bagging, boosting and variants Machine Learning 36 1 1999 105 139
6. J.P. Bickel, Y. Ritov, and A. Zakai Some theory for generalized boosting algorithms Journal of Machine Learning Research 7 2006 705 732
7. A. Blumer, A. Ehrenfeucht, D. Haussler, and M.K. Warmuth OccamÊs razor Information Processing Letter 24 6 1987 377 380
8. L. Breiman Arcing algorithm Annuals of Statistics 26 7 1998 801 849
9. L. Breiman Prediction games and arcing classifiers Neural Computation 11 7 1999 1493 1517
10. L. Breiman Some infinity theory for predictor ensembles Tech. Rep. 577 2000 Statistics Department, University of California Berkeley, CA
11. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone Classification and Regression Trees 1984 Chapman & Hall/CRC Wadsworth
12. 2 loss: Regression and classification Journal of the American Statistical Association 98 462 2003 324 339
13. R. Caruana, and A. Niculescu-Mizil An empirical comparison of supervised learning algorithms Proceeding of 23rd International Conference on Machine Learning Pittsburgh, Pennsylvania 2006 161 168
14. H. Chernoff A measure of asymptotic efficiency of tests of a hypothesis based upon the sum of the observations Annals of Mathematical Statistics 24 4 1952 493 507
15. L. Devroye, L. Györfi, and G. Lugosi A Probabilistic Theory of Pattern Recognition 1996 Springer New York
16. T. Dietterich An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting and randomization Machine Learning 40 2 2000 139 157
17. H. Drucker, and C. Cortes Boosting decision trees D.S. Touretzky, M. Mozer, M.E. Hasselmo, Advances in Neural Information Processing Systems vol. 8 1996 MIT Press Cambridge, MA 479 485
18. Y. Freund, and R.E. Schapire Experiments with a new boosting algorithm Proceeding of 13th International Conference on Machine Learning Bari, Italy 1996 148 156
19. Y. Freund, and R.E. Schapire A decision-theoretic generalization of on-line learning and an application to boosting Journal of Computer and System Sciences 55 1 1997 119 139 (Pubitemid 127433398)
20. J. Friedman, T. Hastie, and R. Tibshirani Additive logistic regression: A statistical view of boosting. with discussions Annuals of Statistics 28 2 2000 337 407 (Pubitemid 33227445)
21. A. Garg, and D. Roth Margin distribution and learning Proceedings of the 20th International Conference on Machine Learning Washington, DC 2003 210 217
22. A.J. Grove, and D. Schuurmans Boosting in the limit: Maximizing the margin of learned ensembles Proceedings of the 15th National Conference on Artificial Intelligence Menlo Park, CA 1998 692 699
23. W. Hoeffding Probability inequalities for sum of bounded random variables Journal of American Statistical Society 58 301 1963 13 30
24. W. Jiang Process consistency for AdaBoost Annuals of Statistics 32 1 2004 13 29 (Pubitemid 41449303)
25. L. Koltchinskii, and D. Panchanko Empirical margin distributions and bounding the generalization error of combined classifiers Annuals of Statistics 30 1 2002 1 50
26. L. Koltchinskii, and D. Panchanko Complexities of convex combinations and bounding the generalization error in classification Annuals of Statistics 33 4 2005 1455 1496
27. G. Lugosi, and N. Vayatis On the bayes-risk consistency of regularized boosting methods Annuals of Statistics 32 1 2004 30 55 (Pubitemid 41449304)
28. L. Mason, J. Baxter, P.L. Bartlett, and M.R. Frean Boosting algorithms as gradient descent S.A. Solla, T.K. Leen, K.-R. Müller, Advances in Neural Information Processing Systems vol. 12 1999 MIT Press Cambridge, MA 512 518
29. A. Maurer Concentration inequalities for functions of independent variables Random Structures and Algorithms 29 2 2006 121 138 (Pubitemid 44342936)
30. A. Maurer, and M. Pontil Empirical Bernstein bounds and sample-variance penalization Proceedings of the 22nd Annual Conference on Learning Theory Montreal, Canada 2009
31. C. McDiarmid On the method of bounded differences Surveys in Combinatorics 1989 Cambridge University Press Cambridge, UK 148 188
32. C. McDiarmid Concentration Probabilistic Methods for Algorithmic Discrete Mathematics 1998 Springer 195 248
33. D. Mease, and A. Wyner Evidence contrary to the statistical view of boosting with discussion Journal of Machine Learning Research 9 2008 131 201
34. I. Mukherjee, C. Rudin, and R. Schapire The rate of convergence of Adaboost Proceedings of the 24th Annual Conference on Learning Theory Budapest, Hungary 2011
35. J.R. Quinlan Bagging, boosting, and C4.5 Proceeding of 13th National Conference on Artificial Intelligence Portland, OR 1996 725 730
36. G. Rätsch, T. Onoda, and K.R. Müller Soft margins for Adaboost Machine Learning 42 3 2001 287 320
37. L. Reyzin, and R.E. Schapire How boosting the margin can also boost classifier complexity Proceeding of 23rd International Conference on Machine Learning Pittsburgh, PA 2006 753 760
38. N. Sauer On the density of families of sets Journal of Combinatorial Theory, Series A 13 1 1972 145 147
39. R. Schapire, Y. Freund, P.L. Bartlett, and W. Lee Boosting the margin: A new explanation for the effectives of voting methods Annuals of Statistics 26 5 1998 1651 1686 (Pubitemid 128376902)
40. J. Shawe-Taylor, and R.C. Williamson Generalization performance of classifiers in terms of observed covering numbers H.U.S.P. Fischer, Proceedings of the 14th European Computational Learning Theory Conference 1999 Springer Berlin 153 167
41. C. Shen, and H. Li Boosting through optimization of margin distributions IEEE Transactions on Neural Networks 21 4 2010 659 666
42. P.K. Shivaswamy, and T. Jebara Variance penalizing adaboost J. Shawe-Taylor, R.S. Zemel, P.L. Bartlett, F.C.N. Pereira, K.Q. Weinberger, Advances in Neural Information Processing Systems vol. 24 2011 MIT Press Cambridge, MA 1908 1916
43. V.N. Vapnik Statistical Learning Theory 1998 John Wiley & Sons New York
44. L.W. Wang, M. Sugiyama, C. Yang, Z.-H. Zhou, and J. Feng A refined margin analysis for boosting algorithms via equilibrium margin Journal of Machine Learning Research 12 2011 1835 1863
45. X. Wu, and V. Kumar The Top Ten Algorithms in Data Mining 2009 Chapman and Hall/CRC
46. T. Zhang Statistical behavior and consistency of classification methods based on convex risk minimization Annals of Statistics 32 1 2004 56 85
47. Z.-H. Zhou Ensemble Methods: Foundations and Algorithms 2012 Chapman & Hall/CRC Boca Raton, FL