Analysis of boosting algorithms using the smooth margin function

Annals of Statistics

Volumn 35, Issue 6, 2007, Pages 2723-2768

  Rudin, Cynthia ^a   Schapire, Robert E ^b,c   Daubechies, Ingrid ^a,b  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b PRINCETON UNIVERSITY   (United States)

^c Princeton University   (United States)

Author keywords

AdaBoost; Arc gv; Boosting; Convergence rates; Coordinate descent; Large margin classification

Indexed keywords

EID: 50849110153     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/009053607000000785     Document Type: Article

References (29)

1. BREIMAN, L. (1998). Arcing classifiers (with discussion). Ann. Statist. 26 801-849. MR 1635406
2. BREIMAN, L. (1999). Prediction games and arcing algorithms. Neural Computation 11 1493-1517.
3. CARUANA, R. and NICULESCU-MIZIL, A. (2006). An empirical comparison of supervised learning algorithms. In Proc. Twenty-Third International Conference on Machine Learning 161-168. ACM Press, New York.
4. COLLINS, M., SCHAPIRE, R. E. and SINGER, Y. (2002). Logistic regression, AdaBoost and Bregman distances. Machine Learning 48 253-285.
5. DRUCKER, H. and CORTES, C. (1996). Boosting decision trees. In Advances in Neural Information Processing Systems 8 479-485. MIT Press, Cambridge, MA.
6. DUFFY, N. and HELMBOLD, D. (1999). A geometric approach to leveraging weak learners. Computational Learning Theory (Nordkirchen, 1999). Lecture Notes in Comput. Sci. 1572 18-33. Springer, Berlin. MR1724977
7. FREUND, Y. and SCHAPIRE, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55 119-139. MR1473055
8. FRIEDMAN, J., HASTIE, T. and TIBSHIRANI, R. (2000). Additive logistic regression: A statistical view of boosting (with discussion). Ann. Statist. 28 337-407. MR1790002
9. GROVE, A. J. and SCHUURMANS, D. (1998). Boosting in the limit: Maximizing the margin of learned ensembles. In Proc. Fifteenth National Conference on Artificial Intelligence 692-699.
10. KOLTCHINSKII, V. and PANCHENKO, D. (2005). Complexities of convex combinations and bounding the generalization error in classification. Ann. Statist. 33 1455-1496. MR2166553
11. KUTIN, S. (2002). Algorithmic stability and ensemble-based learning. Ph.D. dissertation, Univ. Chicago.
12. MASON, L., BAXTER, J., BARTLETT, P. and FREAN, M. (2000). Boosting algorithms as gradient descent. In Advances in Neural Information Processing Systems 12 512-518. MIT Press, Cambridge, MA.
13. MEIR, R. and RÄTSCH, G. (2003). An introduction to boosting and leveraging. Advanced Lectures on Machine Learning. Lecture Notes in Comput. Sci. 2600 119-183. Springer, Berlin.
14. QUINLAN, J. R. (1996). Bagging, boosting, and C4.5. In Proc. Thirteenth National Conference on Artificial Intelligence 725-730. AAAI Press, Menlo Park, CA.
15. RÚTSCH, G. (2001). Robust boosting via convex optimization: Theory and applications. Ph.D. dissertation, Dept. Computer Science, Univ. Potsdam, Potsdam, Germany.
16. RATSCH, G., ONODA, T. and MÜLLER, K.-R. (2001). Soft margins for AdaBoost. Machine Learning 42 287-320.
17. RÄTSCH, G. and WARMUTH, M. (2005). Efficient margin maximizing with boosting. J. Much. Learn. Res. 6 2131-2152. MR2249883
18. REYZIN, L. and SCHAPIRE, R. E. (2006). How boosting the margin can also boost classifier complexity. In Proc. Twenty-Third International Conference on Machine Learning 753-760. ACM Press, New York.
19. ROSSET, S., ZHU, J. and HASTIE, T. (2004). Boosting as a regularized path to a maximum margin classifier. J. Mach. Learn. Res. 5 941-973. MR2248005
20. RUDIN, C. (2004). Boosting, margins and dynamics. Ph.D. dissertation, Princeton Univ.
21. RUDIN, C., CORTES, C., MOHRI, M. and SCHAPIRE, R. E. (2005). Margin-based ranking meets boosting in the middle. Learning Theory. Lecture Notes in Comput. Sci. 3559 63-78. Springer, Berlin. MR2203254
22. RUDIN, C., DAUBECHIES, I. and SCHAPIRE, R. E. (2004). The dynamics of AdaBoost: Cyclic behavior and convergence of margins. J. Mach. Learn. Res. 5 1557-1595. MR2248027
23. RUDIN, C., DAUBECHIES, I. and SCHAPIRE, R. E. (2004). On the dynamics of boosting. In Advances in Neural Information Processing Systems 16. MIT Press, Cambridge, MA.
24. RUDIN, C. and SCHAPIRE, R. E. (2007). Margin-based ranking and why Adaboost is actually a ranking algorithm. To appear.
25. RUDIN, C., SCHAPIRE, R. E. and DAUBECHIES, I. (2004). Boosting based on a smooth margin. Learning Theory. Lecture Notes in Comput. Sci. 3120 502-517. Springer. Berlin. MR2177931
26. RUDIN, C., SCHAPIRE, R. E. and DAUBECHIES, I. (2007). Precise statements of convergence for AdaBoost and arc-gv. In Proc. AMS-IMS-SIAM Joint Summer Research Conference: Machine Learning, Statistics, and Discovery 131-145.
27. SCHAPIRE, R. E. (2003). The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification. Lecture Notes in Statist. 171 149-171. Springer, New York. MR2005788
28. SCHAPIRE, R. E., FREUND, Y., BARTLETT, P. and LEE, W. S. (1998). Boosting the margin: A new explanation for the effectiveness of voting methods. Ann. Statist. 26 1651-1686. MR 1673273
29. ZHANG, T. and YU, B. (2005). Boosting with early stopping: Convergence and consistency. Ann. Statist. 33 1538-1579. MR2166555