On the use of K-fold cross-validation to choose cutoff values and assess the performance of predictive models in stepwise regression

International Journal of Biostatistics

Volumn 5, Issue 1, 2009, Pages

  Mahmood, Zafar ^a   Khan, Salahuddin ^b  

^a THE UNIVERSITY OF AGRICULTURE   (Pakistan)

^b UNIVERSITY OF PESHAWAR   (Pakistan)

Author keywords

Cross validation; Cutoff values; Prediction; Stepwise regression; Variable selection

Indexed keywords

ANALYTICAL ERROR; ARTICLE; AUTOMATION; K FOLD CROSS VALIDATION; METHODOLOGY; MONTE CARLO METHOD; PREDICTION; REGRESSION ANALYSIS; SAMPLING; STATISTICAL MODEL; VALIDATION PROCESS;

EID: 68549112870     PISSN: 15574679     EISSN: None     Source Type: Journal    
DOI: 10.2202/1557-4679.1105     Document Type: Article

References (35)

1. Anderssen E., Dyrstad K., Westad F., and Martens H., Reducing Over-Optimism in Variable Selection by Cross-Model Validation. Chemometrics and Intelligent Lab. Systems 84 (2006) 69-74. (Pubitemid 44708396)
2. Aldrin M., Improved Prediction Penalizing Both Slope and Curvature in Additive Models. Computational Statistics and Data Analysis 50 (2006) 267-284. (Pubitemid 41315784)
3. Baumann K., and Stiefl N., Validation Tools for Variable Subset Regression. J. of Comp.-Aided Molecular Design. 18(2004) 549-562.
4. Burman P., A comparative Study of Ordinary Cross-Validation, V-fold Croos- Validation and the Repeated Learning Testing Methods. Biomerika. 76 (1989) 503-514.
5. Burman P., Estimation of Optimal Transformation Using v-fold Cross-Validation and Repeated Learning -Testing Methods. Sankhya Ser. A 52 (1990) 314-345.
6. Breiman L., Friedman J.H., Olshen R.A., and Stone, Classification and Regression Tree. Wadsworth, Belmont, CA (1984).
7. Breiman, L., Better Subset Regression Using the Nonnegative Garrote. Tchnomatrics 37 (1995) 373-384
8. Bengio Y., and Grandvalet Y., No Unbiased Estimator of the Variance of K-fold Cross-Validation. Journal of Machine Learning Research 5(2004) 1089-1105.
9. Clark R.D., Boosted Leave-Many-Out Cross-Validation: The Effect of Training and Test Set Diversity on PLS Statistics. J. of Comp.-Aided Molecular Design. 17(2003) 265-275.
10. Duan K. et. Al., Evaluating of simple Performance Measures for Tunning SVM Hyperparameters. Neurocomputing 51(2003) 41-59. (Pubitemid 36367224)
11. Davison, A.C., Hinkley, D.V., Bootstrap Method and Their Applications. Cambridge University Press, Cambridge, UK, 1997.
12. Dudoit S. and Van der Laan M.J., Asymptotics of Cross-Validated Risk Estimation in Estimator Selection and Performance Assessment. Statistical Methodology 2(2) (2005) 131-154.
13. Efron B., How Biased is the apparent Error Rate of a Prediction Rule? J. Amer. Statist. Assoc., 81 (1986) 461-470.
14. Geisser, S., The Predictive Sample Reuse Method with Applications, J. Amer. Statist. Assoc., 70 (1975) 320-328.
15. Glantz, S.A. and Sliker, B.K., Primer of Applied Regression and Analysis of Variance, McGraw-Hill, 2000.
16. Herzberg A.M., and Tsukanove A.V., A Note on the Modification of Jackknife Criterion for Model Selection. Utilitas Math. 29 (1986) 209-216.
17. Jonathan P., Krzanowski and McCarthy, On the Use of Cross-Validation to Assess Performance in Multivariate Prediction. Statistics and Computing. 10 (2000) 209-229.
18. Kittler J., Feature Set Search Algorithm. In C.H. Chen, Editor, Pattern Recognition and Signal Processing, Sijthoff & Noordhoff, (1978) 41-60.
19. Kohavi R., A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. International Joint conf. on Artificial Intelligence (IJCAI). 1995.
20. th Editoin) McGraw-Hell/Irvin Series, 2004.
21. Lachenbruch P.A., and Mickey M.R., Estimation of Error Rates in Discriminant Analysis. Technomatrics. 10 (1968) 1-11
22. Pudil P., Novovicova J., and Kittler J., Floating Search Method in Feature Selection. Pattern Recognition Letters, 15(11) (1994) 1119-1125
23. R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3- 900051-07-0; URL http://www.R-project.org, (2004).
24. Snee, R. D., Validation of regression models: methods and examples. Technometrics, 19 (1977) 415-428.
25. Stone, M., Cross-Validatory Choice and Assessment of Statistical Predictions (with discussion). Journal of the Royal Statistical Society, Series B, 36 (1974) 111-147.
26. Salahuddin and Hawkes, A.G., Cross-Validation in Stepwise Regression. Commu. Statist. Theory Meth., 20(4) (1991) 1163-1182.
27. Shao J., Linear Model Selection by Cross-Validation. Am. Stat. Assoc. 88(422) (1993) 486-494.
28. So S.S., Heldon S.P., Geerestein and Karplus M., Quantitative Structure-Activity Relationship Studies of Progesterone Receptor Binding Steroids. J. Chem. Inf. Comput. Sci. 40 (2000) 762-772
29. rd Int. Joint Conf. on Pattern recognition, Coronado, CA, USA, (1976) 71-75.
30. Somol P. and Pudil P., Oscillating Search Algorithms for Feature Selection. In Proc. Of the 15th Int. Conf. on Pattern recognition (ICPR'2000), IEEE Computer Sociaty (2000) 406-409.
31. Somol P., Pudil P., Novovicova J., and Paclik P., Adaptive Floating Search Method in feature Selection. Pattern Recognition Letters, 20(11-13) (1999) 1157-1163. (Pubitemid 32261893)
32. Vehtari A., and Lampinen J., Model Selection via Predictive Explanatory Power. Helsinki Uni. Of Tech, Lab of Computational Engg. Publications. Report B. ISSN (2004) 1455-1474.
33. Van der Laan M.J., Dudoit S., and Kales S., Asymptotic Optimality of Likelihood-Based Cross-Validation. Stat. Appl. Genet. Mol. Biol.3 (1), Art. 4, (2004) pp. (electronic)
34. Van der Vaart , Aad W., Dudoit, S., and van der Laan, M.J., Oracle Inequalities for Multi-Fold Cross-Validation. Statistics and Decisions, 24, No.3, (2006) 351-371
35. Wisnowski, J.W., Simpson, J.R., Montgomery, D.C., Runger, G.C., Resampling Method for Variable Selection in Robust Regression. Comp. Stats. & Data Analysis 43 (2003) 341-355. (Pubitemid 36854733)