An experimental comparison of cross-validation techniques for estimating the area under the ROC curve

Computational Statistics and Data Analysis

Volumn 55, Issue 4, 2011, Pages 1828-1844

  Airola, Antti ^a,b   Pahikkala, Tapio ^a,b   Waegeman, Willem ^c   De Baets, Bernard ^c   Salakoski, Tapio ^a,b  

^a UNIVERSITY OF TURKU   (Finland)

^b TURKU CENTRE FOR COMPUTER SCIENCE   (Finland)

^c GHENT UNIVERSITY   (Belgium)

Author keywords

Area under the ROC curve; Classifier performance estimation; Conditional AUC estimation; Cross validation; Leave pair out cross validation

Indexed keywords

ALTERNATIVE APPROACH; AREA UNDER THE ROC CURVE; CLASSIFICATION PERFORMANCE; CLASSIFIER PERFORMANCE ESTIMATION; CONDITIONAL AUC ESTIMATION; CROSS VALIDATION; CROSS-VALIDATION TECHNIQUE; EFFICIENT ALGORITHM; EXPERIMENTAL COMPARISON; EXTENSIVE SIMULATIONS; LEAST SQUARE; PERFORMANCE MEASURE; PREDICTIVE MODELS; SMALL DATA SET;

ALGORITHMS; CLASSIFICATION (OF INFORMATION); CLASSIFIERS;

ESTIMATION;

EID: 78650847582     PISSN: 01679473     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.csda.2010.11.018     Document Type: Article

References (47)

1. S. Agarwal, T. Graepel, R. Herbrich, S. Har-Peled, and D. Roth Generalization bounds for the area under the ROC curve Journal of Machine Learning Research 6 2005 393 425
2. A. Airola, T. Pahikkala, W. Waegeman, B. De Baets, and T. Salakoski A comparison of AUC estimators in small-sample studies S. Deroski, P. Geurts, J. Rousu, Proceedings of the Third International Workshop on Machine Learning in Systems Biology MLSB'09 2009 Helsinki University Printing House Helsinki, Finland 15 23
3. A. Airola, S. Pyysalo, J. Bjrne, T. Pahikkala, F. Ginter, and T. Salakoski All-paths graph kernel for proteinprotein interaction extraction with evaluation of cross-corpus learning BMC Bioinformatics 9 Suppl. 11 2008 S2
4. S. An, W. Liu, and S. Venkatesh Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression Pattern Recognition 40 8 2007 2154 2162
5. S. Baker, and B. Kramer Identifying genes that contribute most to good classification in microarrays BMC Bioinformatics 7 2006 407
6. A.P. Bradley The use of the area under the ROC curve in the evaluation of machine learning algorithms Pattern Recognition 30 7 1997 1145 1159
7. U.M. Braga-Neto, and E.R. Dougherty Is cross-validation valid for small-sample microarray classification? Bioinformatics 20 3 2004 374 380
8. C. Cortes, and M. Mohri AUC optimization vs. error rate minimization S. Thrun, L. Saul, B. Schlkopf, Advances in Neural Information Processing Systems vol. 16 2004 MIT Press Cambridge, Massachusetts, USA
9. C. Cortes, M. Mohri, and A. Rastogi An alternative ranking problem for search engines C. Demetrescu, Proceedings of the 6th Workshop on Experimental Algorithms Lecture Notes in Computer Science vol. 4525 2007 Springer Berlin, Heidelberg, Germany 1 21
10. T.G. Dietterich Approximate statistical tests for comparing supervised classification learning algorithms Neural Computation 10 1998 1895 1923
11. T. Fawcett, and P.A. Flach A response to Webb and Ting's "On the application of ROC analysis to predict classification performance under varying class distributions" Machine Learning 58 1 2005 33 38
12. G. Forman, and M. Scholz Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement SIGKDD Explorations 12 1 2010 49 57
13. G. Fung, and O.L. Mangasarian Proximal support vector machine classifiers KDD'01: Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2001 ACM New York, NY, USA 77 86
14. O. Gevaert, F. De Smet, D. Timmerman, Y. Moreau, and B. De Moor Predicting the prognosis of breast cancer by integrating clinical and microarray data with Bayesian networks Bioinformatics 22 14 2006 184 190
15. B. Hanczar, J. Hua, C. Sima, J. Weinstein, M. Bittner, and E.R. Dougherty Small-sample precision of ROC-related estimates Bioinformatics 26 6 2010 822 830
16. J.A. Hanley, and B.J. McNeil The meaning and use of the area under a receiver operating characteristic (ROC) curve Radiology 143 1 1982 29 36
17. T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu The entire regularization path for the support vector machine Journal of Machine Learning Research 5 2004 1391 1415
18. T. Hastie, R. Tibshirani, and J. Friedman The Elements of Statistical Learning: Data Mining, Inference and Prediction Second ed. Springer Series in Statistics 2009 Springer
19. R. Herbrich, T. Graepel, and K. Obermayer Support vector learning for ordinal regression Proceedings of the Ninth International Conference on Artificial Neural Networks ICANN'99 1999 Institute of Electrical Engineers London 97 102
20. A.E. Hoerl, and R.W. Kennard Ridge regression: biased estimation for nonorthogonal problems Technometrics 12 1970 55 67
21. J. Huang, and C.X. Ling Using AUC and accuracy in evaluating learning algorithms IEEE Transactions on Knowledge and Data Engineering 17 3 2005 299 310
22. J.-H. Kim Estimating classification error rate: repeated cross-validation, repeated hold-out and bootstrap Computational Statistics & Data Analysis 53 2009 3735 3745
23. R. Kohavi A study of cross-validation and bootstrap for accuracy estimation and model selection C. Mellish, Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence vol. 2 1995 Morgan Kaufmann San Mateo, California, USA 1137 1143
24. W. Krzanowski, and D. Hand Assessing error rate estimators: the leave-one-out method reconsidered Australian Journal of Statistics 39 1 1997 35 46
25. A. Luntz, and V. Brailovsky On estimation of characters obtained in statistical procedure of recognition Technicheskaya Kibernetica 3 1969 563 575
26. M. Miwa, R. Sætre, Y. Miyao, and J. Tsujii Proteinprotein interaction extraction by leveraging multiple kernels and parsers International Journal of Medical Informatics 78 2009 e39 e46
27. Pahikkala, T., Airola, A., Boberg, J., Salakoski, T., 2008. Exact and efficient leave-pair-out cross-validation for ranking RLS. In: Honkela, T., Pll, M., Paukkeri, M.-S., Simula, O. (Eds.), Proceedings of the 2nd International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning. Helsinki University of Technology. pp. 18.
28. T. Pahikkala, J. Boberg, and T. Salakoski Fast n-fold cross-validation for regularized least-squares T. Honkela, T. Raiko, J. Kortela, H. Valpola, Proceedings of the Ninth Scandinavian Conference on Artificial Intelligence 2006 Otamedia Espoo, Finland 83 90
29. T. Pahikkala, S. Pyysalo, J. Boberg, J. Jrvinen, and T. Salakoski Matrix representations, linear transformations, and kernels for disambiguation in natural language Machine Learning 74 2 2009 133 158
30. T. Pahikkala, E. Tsivtsivadze, A. Airola, J. Boberg, and J. Jrvinen An efficient algorithm for learning to rank from preference graphs Machine Learning 75 1 2009 129 165
31. Pahikkala, T., Tsivtsivadze, E., Airola, A., Boberg, J., Salakoski, T., 2007. Learning to rank with pairwise regularized least-squares. In: Joachims, T., Li, H., Liu, T.-Y., Zhai, C. (Eds.), SIGIR 2007 Workshop on Learning to Rank for Information Retrieval. pp. 2733.
32. B.J. Parker, S. Gunter, and J. Bedo Stratification bias in low signal microarray studies BMC Bioinformatics 8 2007 326
33. F.J. Provost, T. Fawcett, and R. Kohavi The case against accuracy estimation for comparing induction algorithms J. Shavlik, Proceedings of the Fifteenth International Conference on Machine Learning 1998 Morgan Kaufmann Publishers Inc. San Francisco, CA, USA 445 453
34. Rifkin, R., 2002. Everything old is new again: a fresh look at historical approaches in machine learning. Ph.D. Thesis. Massachusetts Institute of Technology.
35. C. Saunders, A. Gammerman, and V. Vovk Ridge regression learning algorithm in dual variables Proceedings of the Fifteenth International Conference on Machine Learning 1998 Morgan Kaufmann Publishers Inc. San Francisco, California, USA 515 521
36. R.A. Schiavo, and D.J. Hand Ten more years of error rate research International Statistical Review 68 3 2000 295 310
37. J.A.K. Suykens, and J. Vandewalle Least squares support vector machine classifiers Neural Processing Letters 9 3 1999 293 300
38. J.A. Swets Measuring the accuracy of diagnostic systems Science 240 4857 1988 1285 1293
39. C.H. Teo, S.V. Vishwanathan, A. Smola, and Q.V. Le Bundle methods for regularized risk minimization Journal of Machine Learning Research 11 2010 311 365
40. S. Vanderlooy, and E. Hllermeier A critical analysis of variants of the AUC Machine Learning 72 3 2008 247 262
41. V. Vapnik Estimation of Dependences Based on Empirical Data 1979 Nauka Moscow (in Russian); English translation: Springer, New York, 1982
42. V.N. Vapnik The Nature of Statistical Learning Theory 1995 Springer-Verlag New York, Inc. New York, NY, USA
43. W. Waegeman, B. De Baets, and L. Boullart ROC analysis in ordinal regression learning Pattern Recognition Letters 29 1 2008 1 9
44. F. Wilcoxon Individual comparisons by ranking methods Biometrics 1 1945 80 83
45. T. Zhang On the dual formulation of regularized linear systems with convex risks Machine Learning 46 2002 91 129
46. P. Zhang, and J. Peng SVM vs. regularized least squares classification J. Kittler, M. Petrou, M. Nixon, Proceedings of the 17th International Conference on Pattern Recognition 2004 IEEE Computer Society Washington, DC, USA 176 179
47. P. Zhang, J. Peng, and N. Riedel Finite sample error bound for Parzen windows AAAI'05: Proceedings of the 20th National Conference on Artificial Intelligence 2005 AAAI Press 925 930