Practical ensemble classification error bounds for different operating points

IEEE Transactions on Knowledge and Data Engineering

Volumn 25, Issue 11, 2013, Pages 2590-2601

  Varshney, Kush R ^a   Prenger, Ryan J ^b   Marlatt, Tracy L ^b   Chen, Barry Y ^b   Hanley, William G ^c  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b LAWRENCE LIVERMORE NATIONAL LABORATORY   (United States)

^c EXPONENT INC   (United States)

Author keywords

Cantelli inequality; random forests; receiver operating characteristic; reject option

Indexed keywords

CANTELLI INEQUALITY; CLASSIFICATION WITH A REJECT OPTIONS; MISSED DETECTION PROBABILITIES; PROBABILITY OF FALSE ALARM; PROBABILITY OF MISSED DETECTIONS; RANDOM FORESTS; RECEIVER OPERATING CHARACTERISTICS; REJECT OPTION;

ALARM SYSTEMS; DECISION TREES; ERROR ANALYSIS; PROBABILITY;

LEARNING ALGORITHMS;

EID: 84884795623     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2012.219     Document Type: Article

References (34)

1. L. Breiman, "Bagging Predictors," Machine Learning, vol. 24, no. 2, pp. 123-140, Aug. 1996. (Pubitemid 126724382)
2. Y. Freund and R.E. Schapire, "A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting," J. Computer and System Sciences, vol. 55, no. 1, pp. 119-139, Aug. 1997. (Pubitemid 127433398)
3. L. Breiman, "Arcing Classifiers," Annals of Statistics, vol. 26, no. 3, pp. 801-849, June 1998. (Pubitemid 128450035)
4. Y. Zhang and W.N. Street, "Bagging with Adaptive Costs," IEEE Trans. Knowledge Data Eng., vol. 20, no. 5, pp. 577-588, May 2008.
5. K.R. Varshney, D. Fang, A.R. Heching, A. Mojsilovic, and M. Singh, "Classification of IT Service Tickets for Defect Prevention," Proc. INFORMS Ann. Meeting, Nov. 2011.
6. C.K. Chow, "On Optimum Recognition Error and Reject Tradeoff," IEEE Trans. Information Theory, vol. IT-16, no. 1, pp. 41-46, Jan. 1970.
7. H.L. Van Trees, Detection, Estimation, and Modulation Theory. Wiley, 1968.
8. L. Breiman, "Random Forests," Machine Learning, vol. 45, no. 1, pp. 5-32, Oct. 2001. (Pubitemid 32933532)
9. R.E. Schapire, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods," Annals of Statistics, vol. 26, no. 5, pp. 1651-1686, Oct. 1998. (Pubitemid 128376902)
10. L. Wang, M. Sugiyama, Z. Jing, C. Yang, Z.-H. Zhou, and J. Feng, "A Refined Margin Analysis for Boosting Algorithms via Equilibrium Margin," J. Machine Learning Research, vol. 12, pp. 1835-1863, June 2011.
11. R.J. Prenger, T.D. Lemmond, K.R. Varshney, B.Y. Chen, and W.G. Hanley, "Class-Specific Error Bounds for Ensemble Classifiers," Proc. ACM SIGKDD Conf. Knowledge Discovery and Data Mining, pp. 843-852, July 2010.
12. K.R. Varshney, "A Risk Bound for Ensemble Classification with a Reject Option," Proc. IEEE Statistical Signal Processing Workshop, pp. 773-776, June 2011.
13. H. Kargupta, B.-H. Park, and H. Dutta, "Orthogonal Decision Trees," IEEE Trans. Knowledge Data Eng., vol. 18, no. 8, pp. 1028-1042, Aug. 2006. (Pubitemid 44109533)
14. L.L. Minku, A.P. White, and X. Yao, "The Impact of Diversity on Online Ensemble Learning in the Presence of Concept Drift," IEEE Trans. Knowledge Data Eng., vol. 22, no. 5, pp. 730-742, May 2010.
15. L.L. Minku and X. Yao, "DDD: A New Ensemble Approach for Dealing with Concept Drift," IEEE Trans. Knowledge Data Eng., vol. 24, no. 4, pp. 619-633, Apr. 2012.
16. B. Verma and A. Rahman, "Cluster-Oriented Ensemble Classifier: Impact of Multicluster Characterization on Ensemble Classifier Learning," IEEE Trans. Knowledge Data Eng., vol. 24, no. 4, pp. 605-618, Apr. 2012.
17. N. Usunier, M.-R. Amini, and P. Gallinari, "A Data-Dependent Generalisation Error Bound for the AUC," Proc. ICML Workshop ROC Analysis in Machine Learning, Aug. 2005.
18. R. Herbei and M.H. Wegkamp, "Classification with Reject Option," Canadian J. Statistics, vol. 34, no. 4, pp. 709-721, Dec. 2006.
19. P.L. Bartlett and M.H. Wegkamp, "Classification with a Reject Option Using a Hinge Loss," J. Machine Learning Research, vol. 9, pp. 1823-1840, Aug. 2008.
20. C. Scott, G. Bellala, and R. Willett, "The False Discovery Rate for Statistical Pattern Recognition," Electronic J. Statistics, vol. 3, pp. 651-677, 2009.
21. M. Yuan and M.H. Wegkamp, "Classification Methods with Reject Option Based on Convex Risk Minimization," J. Machine Learning Research, vol. 11, pp. 111-130, Jan. 2010.
22. R. El-Yaniv and Y. Wiener, "On the Foundations of Noise-Free Selective Classification," J. Machine Learning Research, vol. 11, pp. 1605-1641, May 2010.
23. O. Bousquet, "New Approaches to Statistical Learning Theory," Annals Inst. of Statistical Math., vol. 55, no. 2, pp. 371-389, June 2003.
24. Y. Freund, Y. Mansour, and R.E. Schapire, "Generalization Bounds for Averaged Classifiers," Annals of Statistics, vol. 32, no. 4, pp. 1698-1722, Aug. 2004. (Pubitemid 41250282)
25. G. Biau, L. Devroye, and G. Lugosi, "Consistency of Random Forests and Other Averaging Classifiers," J. Machine Learning Research, vol. 9, pp. 2015-2033, Nov. 2008.
26. L.I. Kuncheva and C.J. Whitaker, "Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy," Machine Learning, vol. 51, no. 2, pp. 181-207, May 2003.
27. G.I. Webb and Z. Zheng, "Multistrategy Ensemble Learning: Reducing Error by Combining Ensemble Learning Techniques," IEEE Trans. Knowledge Data Eng., vol. 16, no. 8, pp. 980-991, Aug. 2004.
28. E.K. Tang, P.N. Suganthan, and X. Yao, "An Analysis of Diversity Measures," Machine Learning, vol. 65, no. 1, pp. 247-271, Oct. 2006. (Pubitemid 44451200)
29. S. Wang and X. Yao, "Relationships between Diversity of Classification Ensembles and Single-Class Performance Measures," IEEE Trans. Knowledge Data Eng., vol. 25, no. 1, pp. 206-219, Jan. 2013.
30. T. Pietraszek, "On the Use of ROC Analysis for the Optimization of Abstaining Classifiers," Machine Learning, vol. 68, no. 2, pp. 137-169, Aug. 2007. (Pubitemid 47165757)
31. F.P. Cantelli, "Sui Confini Della Probabilità," Atti del Congresso Internazionale dei Matematici, vol. 6, pp. 47-59, Sept. 1928.
32. D. Bertsimas and I. Popescu, "Optimal Inequalities in Probability Theory: A Convex Optimization Approach," SIAM J. Optimization, vol. 15, no. 3, pp. 780-804, 2005. (Pubitemid 41156915)
33. A. Asuncion and D.J. Newman, "UCI Machine Learning Repository," http://archive.ics.uci.edu/ml, 2007.
34. C. Chen, A. Liaw, and L. Breiman, "Using Random Forest to Learn Imbalanced Data," Technical Report 666, Dept. of Statistics, Univ. of California, Berkeley, July 2004.