Asymptotics of cross-validated risk estimation in estimator selection and performance assessment

Statistical Methodology

Volumn 2, Issue 2, 2005, Pages 131-154

  Dudoit, Sandrine ^a   Van Der Laan, Mark J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Asymptotic linearity; Asymptotic optimality; Classification; Confidence interval; Cross validation; Density estimation; Estimator selection; Generalization error; Indicator loss function; L2 loss function; Loss function; Model selection

Indexed keywords

EID: 26644446561     PISSN: 15723127     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.stamet.2005.02.003     Document Type: Article

References (39)

1. A. Antos, L. Devroye, and L. Györfi Lower bounds for Bayes error estimation IEEE Transactions on Pattern Analysis and Machine Intelligence 21 7 1999 643 645
2. A. Barron, L. Birgé, and P. Massart Risk bounds for model selection via penalization Probability Theory and Related Fields 113 1999 301 413
3. L. Birgé, and P. Massart From model selection to adaptive estimation D. Pollard E. Torgersen G. Yang Festschrift for Lucien Le Cam: Research Papers in Probability and Statistics 1997 Springer-Verlag New York 55 87
4. L. Breiman The little bootstrap and other methods for dimensionality selection in regression: x -fixed prediction error Journal of the American Statistical Association 87 419 1992 738 754
5. L. Breiman Heuristics of instability and stabilization in model selection Annals of Statistics 24 6 1996 2350 2383
6. L. Breiman, Out-of-bag estimation, Technical Report, Department of Statistics, University of California, Berkeley, 1996
7. L. Breiman, J.H. Friedman, R. Olshen, and C.J. Stone Classification and Regression Trees The Wadsworth Statistics/Probability Series 1984 Wadsworth International Group
8. L. Breiman, and P. Spector Submodel selection and evaluation in regression: the x random case International Statistical Review 60 1992 291 319
9. P. Burman A comparative study of ordinary cross-validation, v -fold cross-validation and the repeated learning-testing methods Biometrika 76 1989 503 514
10. L. Devroye, L. Györfi, and G. Lugosi A Probabilistic Theory of Pattern Recognition 1996 Springer-Verlag New York
11. L. Devroye, D. Schäfer, L. Györfi, and H. Walk The estimation problem of minimum mean squared error Statistics and Decisions 21 2003 15 28
12. S. Dudoit, M.J. van der Laan, Asymptotics of cross-validated risk estimation in model selection and performance assessment, Technical Report 126, Division of Biostatistics, University of California, Berkeley, 2003. URL: http://www.bepress.com/ucbbiostat/paper126
13. S. Dudoit, M.J. van der Laan, S. Keleş, A.M. Molinaro, S.E. Sinisi, and S.L. Teng Loss-based estimation with cross-validation: Applications to microarray data analysis G. Piatetsky-Shapiro P. Tamayo Microarray Data Mining SIGKDD Explorations 5 2 2003 56 68 ACM
14. B. Efron Estimating the error rate of a prediction rule: Improvement on cross-validation Journal of the American Statistical Association 78 1983 316 331
15. B. Efron, and R.J. Tibshirani An Introduction to the Bootstrap 1993 Chapman and Hall/CRC
16. S. Geisser The predictive sample reuse method with applications Journal of the American Statistical Association 70 1975 320 328
17. G. Gong Cross-validation, the jackknife, and the bootstrap: Excess error estimation in forward logistic regression Journal of the American Statistical Association 81 1986 108 113
18. L. Györfi, and M. Horváth On the asymptotic normality of a resubstitution error estimate A. Rizzi M. Vichi H.H. Bock Advances in Data Science and Classification 1998 Springer 197 204
19. L. Györfi, M. Kohler, A. Krzyżak, and H. Walk A Distribution-Free Theory of Nonparametric Regression 2002 Springer-Verlag New York
20. L. Györfi, D. Schäfer, and H. Walk Relative stability of global errors of nonparametric function estimators IEEE Transactions on Information Theory 48 8 2002 2230 2242
21. T. Hastie, R. Tibshirani, and J.H. Friedman The Elements of Statistical Learning: Data Mining, Inference, and Prediction 2001 Springer-Verlag
22. S. Keleş, M.J. van der Laan, and S. Dudoit Asymptotically optimal model selection method with right censored outcomes Bernoulli 10 6 2004 1011 1037
23. K.-C. Li Asymptotic optimality for c p, c l, cross-validation and generalized cross-validation: Discrete index set Annals of Statistics 15 1987 958 975
24. P.J. McCarthy The use of balanced half-sample replication in cross-validation studies Journal of the American Statistical Association 71 1976 596 604
25. A.M. Molinaro, S. Dudoit, and M.J. van der Laan Tree-based multivariate regression and density estimation with right-censored data S. Dudoit R.C. Gentleman M.J. van der Laan Multivariate Methods in Genomic Data Analysis Journal of Multivariate Analysis 90 1 2004 154 177 Elsevier
26. A.M. Molinaro, M.J. van der Laan, Deletion/Substitution/Addition algorithm for partitioning the covariate space in prediction, Technical Report 162, Division of Biostatistics, University of California, Berkeley, 2004. URL: http://www.bepress.com/ucbbiostat/paper162
27. R.R. Picard, and R.D. Cook Cross-validation of regression models Journal of the American Statistical Association 79 1984 575 583
28. M. Pintér On the rate of convergence of error estimates for the partitioning classification rule Theoretical Computer Science 284 1 2002 181 196
29. B.D. Ripley Pattern Recognition and Neural Networks 1996 Cambridge University Press Cambridge, New York
30. J. Shao Linear model selection by cross-validation Journal of the American Statistical Association 88 1993 486 494
31. S.E. Sinisi, and M.J. van der Laan Deletion/substitution/addition algorithm in learning with applications in genomics Statistical Applications in Genetics and Molecular Biology 3 1 2004 (Article 18). URL: http://www.bepress.com/sagmb/vol3/iss1/art18
32. M. Stone Cross-validatory choice and assessment of statistics predictions Journal of the Royal Statistical Society, Series B 36 2 1974 111 147
33. M. Stone Asymptotics for and against cross-validation Biometrika 64 1 1977 29 35
34. M.J. van der Laan, S. Dudoit, Unified cross-validation methodology for selection among estimators and a general cross-validated adaptive ε -net estimator: Finite sample oracle inequalities and examples, Technical Report 130, Division of Biostatistics, University of California, Berkeley, 2003. URL: http://www.bepress.com/ucbbiostat/paper130
35. M.J. van der Laan, S. Dudoit, and S. Keleş Asymptotic optimality of likelihood-based cross-validation Statistical Applications in Genetics and Molecular Biology 3 1 2004 (Article 4). URL: http://www.bepress.com/sagmb/vol3/iss1/art4
36. M.J. van der Laan, S Dudoit, A.W. van de Vaart, The cross-validated adaptive ε -net estimator, Technical Report 142, Division of Biostatistics, University of California, Berkeley, 2004. URL: http://www.bepress.com/ucbbiostat/paper142
37. M.J. van der Laan, and J.M. Robins Unified Methods for Censored Longitudinal Data and Causality 2003 Springer-Verlag New York
38. A.W. van der Vaart, and J. Wellner Weak Convergence and Empirical Processes 1996 Springer-Verlag New York
39. P. Zhang Model selection via multifold cross-validation Annals of Statistics 21 1993 299 313