Analysis of variance of cross-validation estimators of the generalization error

Journal of Machine Learning Research

Volumn 6, Issue , 2005, Pages

  Markatou, Marianthi ^a   Tian, Hong ^a   Biswas, Shameek ^a   Hripcsak, George ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

Cross validation; Generalization error; Moment approximation; Prediction; Variance estimation

Indexed keywords

APPROXIMATION THEORY; DATA REDUCTION; DATABASE SYSTEMS; INFORMATION ANALYSIS; PROBLEM SOLVING;

CROSS-VALIDATION; GENERALIZATION ERROR; MOMENT APPROXIMATION; PREDICTION; VARIANCE ESTIMATION;

ERROR ANALYSIS;

EID: 23244454640     PISSN: 15337928     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (26)

1. Y. Bengio and Y. Grandvalet. No unbiased estimator of the variance of k-fold cross validation. Journal of Machine Learning Research, 5: 1089-1105, 2004.
2. P. J. Bickel and K. A. Doksum. Mathematical Statistics, Prentice Hall, 2001.
3. L. Breiman. Heuristics of instability and stabilization in model selection. The Annals of Statistics, 24: 2350-2383, 1996.
4. H. Cramer. Mathematical Methods of Statistics. Princeton University Press, 19th Printing, 1999.
5. T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10: 1895-1923, 1998.
6. B. Efron. Estimating the error rate of a predication rule: Improvement on cross-validation. Journal of the American Statistical Association, 78: 316-331, 1983.
7. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, 1993.
8. B. Efron. The Estimation of Prediction Error: Covariance Penalties and Cross-Validation. Journal of the American Statistical Association, 99: 619-632, 2004.
9. T. Hastie, R. Tibshirani and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer-Verlag, 2001.
10. K. Hitomi and M. Kagihara. Calculation methods for nonlinear dynamic least absolute deviations estimator. Journal of the Japan. Statist. Society, 31: 39-51, 2001.
11. A. D. Ioffe and J-P. Penot. Limiting subhessians, limiting subjets and their calculus. Transactions of the American Mathematical Society, 349: 789-807, 1997.
12. G. M. James. Variance and bias for general loss functions. Machine Learning, 51: 115-135, 2003.
13. M. Kearns. A bound on the error of cross validation with consequences for the training-test split. In Advances in Neural Information Processing Systems, 8: 183-189, 1996.
14. M. Kearns and D. Ron. Algorithmic stability and sanity-check bounds for leave-one-out cross validation. Neural Computation, 11: 1427-1453, 1999.
15. R. A. Khan. Approximation of the expectation of a function of the sample mean. Statistics, 38: 117-122, 2004.
16. R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In The International Joint Conference on Artificial Intelligence, 1137-1143, 1995.
17. E. L. Lehmann. Theory of Point Estimation. Wiley and Sons, 1983.
18. G. J. McLachlan. An asymptotic expansion for the variance of the errors of misclassification of the linear discriminant function. Australian Journal of Statistics, 14: 68-72, 1972.
19. G. J. Mclachlan. An asymptotic expansion of the expectation of the estimated error rate in discriminant analysis. Australian Journal of Statistics, 15: 210-214, 1974.
20. G. J. McLachlan. The asymptotic distributions of the conditional error rate and risk in discriminant analysis. Biometrika, 61: 131-135, 1974.
21. G. J. McLachlan. The bias of the apparent error rate in discriminant analysis. Biometrika, 63: 239-244, 1976.
22. C. Nadeau and Y. Bengio. Inference for the generalization error. Machine Learning, 52: 239-281, 2003.
23. R. R. Picard and R. D. Cook. Cross validation of regression models. Journal of the American Statistical Association, 79: 575-583, 1984.
24. J. Piper. Variability and bias in experimentally measured classifier error rates. Pattern Recognition Letters, 13: 685-692, 1992.
25. E. Ronchetti and L. Ventura. Between stability and higher order asymptotics .Statistics and Computing, 11: 67-73, 2001.
26. P. K. Sen and J. M. Singer. Large Sample Methods in Statistics: An Introduction with Applications. Chapman and Hall, 1993.