No unbiased estimator of the variance of K-fold cross-validation

Journal of Machine Learning Research

Volumn 5, Issue , 2004, Pages 1089-1105

  Bengio, Yoshua ^a   Grandvalet, Yves ^b  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

^b UNIVERSITÉ DE TECHNOLOGIE DE COMPIÈGNE   (France)

Author keywords

Cross validation; K fold cross validation; Statistical comparisons of algorithms; Variance estimators

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COVARIANCE MATRIX; DEGREES OF FREEDOM (MECHANICS); EIGENVALUES AND EIGENFUNCTIONS; LEARNING SYSTEMS;

CROSS VALIDATION; GENERALIZATION PERFORMANCE; K FOLD CROSS VALIDATIONS; NUMERICAL EXPERIMENTS; QUANTITATIVE EXPERIMENTS; STATISTICAL COMPARISONS; THREE DEGREES OF FREEDOM; VARIANCE ESTIMATORS;

UNCERTAINTY ANALYSIS;

EID: 84925604888     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (13)

1. E. Alpaydin. Combined 5×2 cv F test for comparing supervised classification learning algorithms. Neural Computation, 11(8):1885-1892, 1999.
2. M. Anthony and S. B. Holden. Cross-validation for binary classification by real-valued functions: Theoretical analysis. In Proceedings of the International Conference on Computational Learning Theory, pages 218-229, 1998.
3. A. Blum, A. Kalai, and J. Langford. Beating the hold-out: Bounds for k-fold and progressive crossvalidation. In Proceedings of the International Conference on Computational Learning Theory, pages 203-208, 1999.
4. L. Breiman. Heuristics of instability and stabilization in model selection. The Annals of Statistics, 24(6):2350-2383, 1996.
5. A. P. Dawid. Prequential analysis. In S. Kotz, C. B. Read, and D. L. Banks, editors, Encyclopedia of Statistical Sciences, Update Volume 1, pages 464-470. Wiley-Interscience, 1997.
6. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996.
7. T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895-1924, 1999.
8. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap, volume 57 of Monographs on Statistics and Applied Probability. Chapman & Hall, 1993.
9. T. J. Hastie and R. J. Tibshirani. Generalized Additive Models, volume 43 of Monographs on Statistics and Applied Probability. Chapman & Hall, 1990.
10. M. Kearns and D. Ron. Algorithmic stability and sanity-check bounds for leave-one-out crossvalidation. Neural Computation, 11(6):1427-1453, 1996.
11. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1137-1143, 1995.
12. C. Nadeau and Y. Bengio. Inference for the generalization error. Machine Learning, 52(3):239-281, 2003.
13. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, B, 36(1):111-147, 1974.