Estimating the confidence interval for prediction errors of support vector machine classifiers

Journal of Machine Learning Research

Volumn 9, Issue , 2008, Pages 521-540

  Jiang, Bo ^a   Zhang, Xuegong ^a   Cai, Tianxi ^b  

^a TSINGHUA UNIVERSITY   (China)

^b HARVARD UNIVERSITY   (United States)

Author keywords

K fold cross validation; Model evaluation; Perturbation resampling; Prediction errors; Support vector machine

Indexed keywords

COMPUTER SIMULATION; ERROR ANALYSIS; MATHEMATICAL MODELS; SUPPORT VECTOR MACHINES;

K-FOLD CROSS-VALIDATION; MODEL EVALUATION; PERTURBATION-RESAMPLING; PREDICTION ERRORS;

PARAMETER ESTIMATION;

EID: 41549142154     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (39)

1. L. Beerenwinkel, B. Schmidt, H. Walter, R. Kaiser, T. Lengauer, D. Hoffmann, K. Korn, and J. Selbig. Diversity and complexity of HIV-1 drug resistance: A bioinformatics approach to predicting phenotype from genotype. Proceedings of the National Academy of Sciences U.S.A., 99:8271-8276, 2002.
2. Y. Bengio and Y. Grandvalet. No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5:1089-1105, 2004.
3. O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
4. T. Cai, L. Tian, and L. J. Wei. Semiparametric Box-Cox power transformation models for censored survival observations. Biometrika, 92:619-632, 2005.
5. R. E. Campo, J. N. Moreno, G. Suarez, N. Miller, M. A. Kolber, D. J. Holder, M. Shivaprakash, D. M. DeAngelis, J. L. Wright, W. A. Schleif, E. A. Emini, and J. H. Condra. Efficacy of indinavir-ritonavir-based regimens in HIV-1-infected patients with prior protease inhibitor failures. AIDS, 17:1933-1939, 2003.
6. C. C. Chang and C. H. Lin. Libsvm - a library for support vector machine. Available at http://www.csie.ntu.edu.tw/cjlin/libsvm/, 2001.
7. T. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10:1895-1923, 1998.
8. R. M. Dudley. Uniform Central Limit Theorems. Cambridge, U.K.: Cambridge University Press, 1999.
9. B. Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7:1-26,1979.
10. B. Efron. How biased is the apparent error rate of a prediction rule. Journal of the American Statistical Association, 81:461-470, 1986.
11. B. Efron. Better bootstrap confidence intervals. Journal of the American Statistical Association, 82: 171-185, 1987.
12. B. Efron and R. Tibshirani. Cross-validation and the bootstrap: Estimating the error rate of a prediction rule. Technical report, Stanford University, 1995.
13. B. Efron and R. Tibshirani. Improvements on cross-validation: The .632+ bootstrap method. Journal of the American Statistical Association, 92:548-560, 1997.
14. W. J. Fu, R. J. Carroll, and S. Wang. Estimating misclassification error with small samples via bootstrap cross-validation. Bioinformatics, 21:1979-1986, 2005.
15. P. Hall and Y Maesono. A weighted bootstrap approach to bootstrap iteration. Journal of the Royal Statistical Society Series B, 62:137-144, 2000.
16. P. Hall and E. Mammen. On general resampling algorithms and their performance in distribution estimation. The Annals of Statistics, 22:2011-2030, 1994.
17. N. L. Hjort and D. Pollard. Asymptotics for minimizers of convex processes. Technical report, Yale University, 1993.
18. M. Kearns and D. Ron. Algorithmic stability and sanity check bounds for leave-one-out crossvalidation. Neural Computation, 11:1427-1453, 1999.
19. R. Y Liu. Bootstrap procedures under some non-i.i.d. models. The Annals of Statistics, 16:16961708, 1988.
20. S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K. R. Müller. Fisher discriminant analysis with kernels. In Neural Networks for Signal Processing IX, pages 41-48, 1999.
21. A. Molinaro, R. Simon, and R. Pfeiffer. Prediction error estimation: A comparison of resampling methods. Bioinformatics, 21:3301-3307, 2005.
22. C. Nadeau and Y Bengio. Inference for the generalization error. Machine Learning, 52:239-281, 2003.
23. W. Newey and D. McFadden. Large sample estimation and hypothesis testing. In D. McFadden and R. Engler, editors, Handbook of Econometrics IV, pages 2113-2245. Amsterdam: North Holland, 1994.
24. M. F. Para, D. V. Glidden, R. Coombs, A. Collier, J. Condra, C. Craig, R. Basse«, R. Leavitt, S. Snyder, V. J. McAuliffe, and C. Boucher. Baseline human immunodeficiency virus type I phenotype, genotype, and RNA response after switching from long-term hard-capsule saquinavir to indinavir or soft-gel-capsule saquinavir in AIDS clinical trials group protocol 333. Journal of Infectious Diseases, 182:733-743, 2000.
25. Y. Park and L. J. Wei. Estimating subject-specific survival functions under the accelerated failure time model. Biometrika, 90:717-723, 2003.
26. D. Pollard. Empirical Process: Theory and Applications. Hay ward, CA: Institute of Mathematical Statistics, 1990.
27. D. Pollard. Asymptotics for least absolute deviation regression estimators. Econometric Theory, 7: 186-199, 1991.
28. J. Reunanen. Overriding in making comparisons between variable selection methods. Journal of Machine Learning Research, 3:1371-1382, 2003.
29. S. Y. Rhee, M. J. Gonzales, R. Kantor, B. J. Betts, J. Ravela, and R. W. Shafer. Human immunodeficiency virus reverse transcriptase and protease sequence database. Nucleic Acids Research, 31: 298-303, 2003.
30. D. Rubin. The bayesian bootstrap. The Annals of Statistics, 9:130-134, 1981.
31. A. J. Saah, D. W. Haas, M. J. DiNubile, J. Chen, D. J. Holder, R. R. Rhodes, M. Shivaprakash, K. K. Bakshi, R. M. Danovich, D. J. Graham, and J. H. Condra. Treatment with indinavir, efavirenz, and adefovir after failure of nelfinavir therapy. Journal of Infectious Diseases, 187:1157-1162, 2003.
32. J. Shao. Bootstrap model selection. Journal of the American Statistical Association, 91:655-665, 1996.
33. N. Shulman, A. Zolopa, D. Havlir, A. Hsu, C. Renz, S. Boller, P. Jiang, R. Rode, J. Gallant, E. Race, D. J. Kempf, and E. Sun. Virtual inhibitory quotient predicts response to ritonavir boosting of indinavir-based therapy in human immunodeficiency virus-infected patients with ongoing viremia. Antimicrobial Agents and Chemotherapy, 146:3907-3916, 2002.
34. I. Steinwart. Support vector machines are universally consistent. Journal of Complexity, 18:768-779, 2002.
35. A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. New York: Springer-Verlag Inc., 1996.
36. V. N. Vapnik. The Nature of Statistical Learning Theory. New York: Springer, 1995.
37. V. N. Vapnik. Statistical Learning Theory. New York: John Wiley and Sons Inc., 1998.
38. S. Varma and R. Simon. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7:91, 2006.
39. C. F. J. Wu. Jackknife, bootstrap, and other resampling methods in regression analysis (with discussion). The Annals of Statistics, 14:1261-1295, 1986.