Overlaying Classifiers: A Practical Approach to Optimal Scoring

Constructive Approximation

Volumn 32, Issue 3, 2010, Pages 619-648

  Clémençon, Stéphan ^a   Vayatis, Nicolas ^b  

^a TELECOM PARISTECH   (France)

^b Paris Descartes University   (France)

Author keywords

AUC criterion; Bipartite ranking; Density level set; Minimum volume set estimation; Piecewise linear approximation; ROC curve; Scoring function; Statistical learning; Sup norm

Indexed keywords

EID: 77957300985     PISSN: 01764276     EISSN: 14320940     Source Type: Journal    
DOI: 10.1007/s00365-010-9084-9     Document Type: Article

References (35)

1. Agarwal, S., Graepel, T., Herbrich, R., Har-Peled, S., Roth, D.: Generalization bounds for the area under the ROC curve. J. Mach. Learn. Res. 6, 393-425 (2005).
2. Audibert, J.-Y., Tsybakov, A.: Fast learning rates for plug-in classifiers. Ann. Stat. 35(2), 608-633 (2007).
3. Bach, F. R., Heckerman, D., Horvitz, E.: Considering cost asymmetry in learning classifiers. J. Mach. Learn. Res. 7, 1713-1741 (2006).
4. Barreno, M., Cardenas, A. A., Tygar, J. D.: Optimal ROC curve for a combination of classifiers. In: NIPS'07 (2007).
5. Boucheron, S., Bousquet, O., Lugosi, G.: Theory of classification: a survey of some recent advances. ESAIM: Probab. Stat. 9, 323-375 (2005).
6. Cavalier, L.: Nonparametric estimation of regression level sets. Statistics 29, 131-160 (1997).
7. Clémençon, S., Lugosi, G., Vayatis, N.: Ranking and scoring using empirical risk minimization. In: Auer, P., Meir, R. (eds.) Proceedings of COLT 2005. Lecture Notes in Computer Science, vol. 3559, pp. 1-15. Springer, Berlin (2005).
8. Clémençon, S., Lugosi, G., Vayatis, N.: Ranking and empirical risk minimization of U-statistics. Ann. Stat. 36(2), 844-874 (2008).
9. Clémençon, S., Vayatis, N.: Ranking the best instances. J. Mach. Learn. Res. 8, 2671-2699 (2007).
10. Clémençon, S., Vayatis, N.: Approximation of the optimal ROC curve and a tree-based ranking algorithm. In: Proceedings of ALT'08 (2008).
11. Clémençon, S., Vayatis, N.: Empirical performance maximization for linear rank statistics. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems 21. Proceedings of NIPS'08, pp. 305-312 (2009).
12. Clémençon, S., Vayatis, N.: Nonparametric estimation of the precision-recall curve. In: Proceedings of the 26th International Conference on Machine Learning, ICML (2009).
13. Clémençon, S., Vayatis, N.: On partitioning rules for bipartite ranking. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS), JMLR: W& CP 5, pp. 97-104 (2009).
14. Clémençon, S., Vayatis, N.: Tree-based ranking methods. IEEE Trans. Inf. Theory 55(9), 4316-4336 (2009).
15. Cortes, C., Mohri, M.: AUC optimization vs. error rate minimization. In: Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems, vol. 16. MIT Press, Cambridge (2004).
16. Devore, R., Lorentz, G.: Constructive Approximation. Springer, Berlin (1993).
17. Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, Berlin (1996).
18. Dudley, R. M.: Uniform Central Limit Theorems. Cambridge University Press, Cambridge (1999).
19. Egan, J. P.: Signal Detection Theory and ROC Analysis. Academic Press, San Diego (1975).
20. Freund, Y., Iyer, R. D., Schapire, R. E., Singer, Y.: An efficient boosting algorithm for combining preferences. J. Mach. Learn. Res. 4, 933-969 (2003).
21. Györfi, L., Köhler, M., Krzyzak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, Berlin (2002).
22. Hanley, J. A., McNeil, J.: The meaning and use of the area under a ROC curve. Radiology 143, 29-36 (1982).
23. Rakotomamonjy, A.: Optimizing area under ROC curve with svms. In: Proceedings of the First Workshop on ROC Analysis in AI (2004).
24. Rigollet, P., Vert, R.: Fast rates for plug-in estimators of density level sets. Technical Report (2006). arXiv: math/0611473v2.
25. Rudin, C.: Ranking with a P-norm push. In: Simon, H. U., Lugosi, G. (eds.) Proceedings of COLT 2006. Lecture Notes in Computer Science, vol. 4005, pp. 589-604. Springer, Berlin (2006).
26. Scott, C.: Performance measures for Neyman-Pearson classification. Technical report, Department of Statistics, Rice University (2005).
27. Scott, C., Bellala, G., Willett, R.: Generalization error analysis for FDR controlled classification. Technical report, Department of Electrical Engineering and Computer Science, University of Michigan (2008).
28. Scott, C., Nowak, R.: A Neyman-Pearson approach to statistical learning. IEEE Trans. Inf. Theory 51(11), 3806-3819 (2005).
29. Scott, C., Nowak, R.: Learning minimum volume sets. J. Mach. Learn. Res. 7, 665-704 (2006).
30. Tsybakov, A.: On nonparametric estimation of density level sets. Ann. Stat. 25(3), 948-969 (1997).
31. Tsybakov, A.: Optimal aggregation of classifiers in statistical learning. Ann. Stat. 32(1), 135-166 (2004).
32. Usunier, N., Amini, M., Gallinari, P.: A data-dependent generalisation error bound for the AUC. In: ROCML ICML 2005 Workshop (2005).
33. van Trees, H. L.: Detection, Estimation, and Modulation Theory, Part I. Wiley, New York (1968).
34. Willett, R., Nowak, R.: Minimax optimal level set estimation. IEEE Trans. Image Process. 16(12), 2965-2979 (2007).
35. Yan, L., Dodier, R. H., Mozer, M., Wolniewicz, R. H.: Optimizing classifier performance via an approximation to the Wilcoxon-Mann-Whitney statistic. In: Fawcett, T., Mishra, N. (eds.) Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 848-855 (2003).