Explaining the success of adaboost and random forests as interpolating classifiers

Journal of Machine Learning Research

Volumn 18, Issue , 2017, Pages 1-33

  Wyner, Abraham J ^a   Olson, Matthew ^a   Bleich, Justin ^a   Mease, David ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b APPLE INC   (United States)

Author keywords

AdaBoost; Classification; Overfitting; Random forests; Tree ensembles

Indexed keywords

CLASSIFICATION (OF INFORMATION); DECISION TREES; FORESTRY; INTERPOLATION; OPTIMIZATION;

BOOSTING ALGORITHM; DIRECT OPTIMIZATION; INTERPOLATING ALGORITHM; LIKELIHOOD FUNCTIONS; OVERFITTING; PREDICTIVE ACCURACY; RANDOM FORESTS; TREE ENSEMBLES;

ADAPTIVE BOOSTING;

EID: 85021054378     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (32)

1. Yali Amit, Gilles Blanchard, and Kenneth Wilder. Multiple randomized classifiers: Mrcl. Technical report, 2000.
2. Joshua Belanich and Luis E Ortiz. On the convergence properties of optimal adaboost. arXiv preprint arXiv: 1212.1108, 2012.
3. Peter J. Bickel, Ya'acov Ritov, and Alon Zakai. Some theory for generalized boosting algorithms. The Journal of Machine Learning Research, 7: 705-732, 2006.
4. Leo Breiman. Bagging predictors. Machine Learning, 24(2): 123-140, 1996.
5. Leo Breiman. Arcing classifier (with discussion and a rejoinder by the author). The Annals of Statistics, 26(3): 801-849, 1998.
6. Leo Breiman. Prediction games and arcing algorithms. Neural computation, 11(7): 1493-1517, 1999.
7. Leo Breiman. Random forests. Machine Learning, 45: 5-32, 2001.
8. Peter Buhlmann and Bin Yu. Boosting with the l 2 loss: Regression and classification. Journal of the American Statistical Association, 98(462): 324-339, 2003.
9. Corinna Cortes, Mehryar Mohri, and Umar Syed. Deep boosting. In Proceedings of the Thirty-First International Conference on Machine Learning (ICML 2014), 2014.
10. Thomas M Cover and Peter E Hart. Nearest neighbor pattern classification. Information Theory, IEEE Transactions on, 13(1): 21-27, 1967.
11. Robert M. Freund, Paul Grigas, and Rahul Mazumder. Adaboost and forward stagewise regression are first-order convex optimization methods. arXiv preprint arXiv: 1307.1192, 2013.
12. Y. Freund and R.E. Shapire. Discussion of additive logistic regression: A statistical view of boosting. Annals of Statistics, 28: 337-374, 2000.
13. Yoav Freund and Robert E. Schapire. Experiments with a new boosting algorithm. In ICML, volume 96, pages 148-156, 1996.
14. Jerome Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, pages 1189-1232, 2001.
15. Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. Additive logistic regression: A statistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics, 28(2): 337-407, 2000.
16. Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artifficial Intelligence, 203: 1-18, 2013.
17. Carlos Guestrin. Pac-learning, vc dimension and margin-based bounds. Machine Learning, 10701: 15781, 2006.
18. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learn- ing, volume 2. Springer, 2009.
19. Wenxin Jiang. On weak base hypotheses and their implications for boosting regression and classification. Annals of statistics, pages 51-73, 2002.
20. Andy Liaw and Matthew Wiener. Classification and regression by randomforest. R news, 2(3): 18-22, 2002.
21. Senen Barro Manuel Fernandez-Delgado, Eva Cernadas and Dinani Amorim. Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1): 3133-3181, 2014.
22. David Mease and Abraham Wyner. Evidence contrary to the statistical view of boosting. The Journal of Machine Learning Research, 9: 131-156, 2008.
23. Indraneel Mukherjee, Cynthia Rudin, and Robert E Schapire. The rate of convergence of adaboost. The Journal of Machine Learning Research, 14(1): 2315-2347, 2013.
24. Alex Niculescu-Mizil and Rich Caruana. An empirical comparison of supervised learning algorithms using different performance metrics. Technical report, Cornell University, 2005.
25. Greg Ridgeway. Generalized boosted regression models. Documentation on the R Package gbm, version 1- 5, 7, 2006.
26. Robert E. Schapire. Explaining adaboost. In Empirical Inference, pages 37-52. Springer, 2013.
27. Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. MIT press, 2012.
28. Robert E. Schapire, Yoav Freund, Peter Bartlett, and Wee Sun Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. Annals of statistics, pages 1651-1686, 1998.
29. Terry M. Therneau and Elizabeth J. Atkinson. An introduction to recursive partitioning using the rpart routines. Technical report, Technical Report 61. URL http: //www. mayo. edu/hsr/techrpt/61. pdf, 1997.
30. Stefan Wager and Guenther Walther. Uniform convergence of random forests via adaptive concentration. arXiv preprint arXiv: 1503.06388, 2015.
31. Liwei Wang, Masashi Sugiyama, Zhaoxiang Jing, Cheng Yang, Zhi-Hua Zhou, and Jufu Feng. A refined margin analysis for boosting algorithms via equilibrium margin. The Journal of Machine Learning Research, 12: 1835-1863, 2011.
32. Abraham J. Wyner. On boosting and the exponential loss. In Proceedings of the Ninth Annual Conference on AI and Statistics Jan, pages 3-6. Citeseer, 2003.