Study on the impact of partition-induced dataset shift on k-fold cross-validation

IEEE Transactions on Neural Networks and Learning Systems

Volumn 23, Issue 8, 2012, Pages 1304-1312

  Moreno Torres, Jose García ^a   Saez, José A ^a   Herrera, Francisco ^a  

^a UNIVERSITY OF GRANADA   (Spain)

Author keywords

Covariate shift; cross validation; dataset shift; partitioning

Indexed keywords

CLASSIFIER PERFORMANCE; COVARIATE SHIFTS; CROSS VALIDATION; DATASET SHIFTS; K FOLD CROSS VALIDATIONS; PARTITIONING; PERFORMANCE ESTIMATION; STABLE PERFORMANCE;

ARTIFICIAL INTELLIGENCE; COMPUTER NETWORKS;

ESTIMATION;

EID: 84876917722     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2199516     Document Type: Article

References (27)

1. R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," in Proc. Int. Joint Conf. Artif. Intell., 1995, pp. 1137-1145.
2. M. Stone, "Asymptotics for and against cross-validation," Biometrika, vol. 64, no. 1, pp. 29-35, 1977.
3. B. Efron and G. Gong, "A leisurely look at the bootstrap, the jackknife, and cross-validation," Amer. Stat., vol. 37, no. 1, pp. 36-48, 1983.
4. J. Zhang, X. Wang, U. Krüger, and F.-Y. Wang, "Principal curve algorithms for partitioning high-dimensional data spaces," IEEE Trans. Neural Netw., vol. 22, no. 3, pp. 367-380, Mar. 2011.
5. H. Zhang, L. Cui, X. Zhang, and Y. Luo, "Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 2226-2236, Dec. 2011.
6. H. He, S. Chen, K. Li, and X. Xu, "Incremental learning from stream data," IEEE Trans. Neural Netw., vol. 22, no. 12, pp. 1901-1914, Dec. 2011.
7. J. Q. Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset Shift in Machine Learning. Cambridge, MA: MIT Press, 2009.
8. J. G. Moreno-Torres, T. Raeder, R. Aláiz-Rodríguez, N. V. Chawla, and F. Herrera, "A unifying view on dataset shift in classification," Pattern Recognit., vol. 45, no. 1, pp. 521-530, Jan. 2012.
9. J. R. Cano, F. Herrera, and M. Lozano, "Evolutionary stratified training set selection for extracting classification rules with trade off precision-interpretability," Data Knowl. Eng., vol. 60, pp. 90-108, Jan. 2007.
10. X. Zeng and T. R. Martinez, "Distribution-balanced stratified crossvalidation for accuracy estimation," J. Experim. Theor. Artif. Intell., vol. 12, no. 1, pp. 1-12, 2000.
11. M. Sugiyama, M. Krauledat, and K.-R. Müller, "Covariate shift adaptation by importance weighted cross validation," J. Mach. Learn. Res., vol. 8, pp. 985-1005, Dec. 2007.
12. A. Storkey, "When training and test sets are different: Characterizing learning transfer," in Dataset Shift in Machine Learning, J. Quiñonero Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Eds. Cambridge, MA: MIT Press, 2009, pp. 3-28.
13. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. San Mateo, CA: Morgan Kaufmann, 2005.
14. H. Shimodaira, "Improving predictive inference under covariate shift by weighting the log-likelihood function," J. Stat. Plann. Inference, vol. 90, no. 2, pp. 227-244, 2000.
15. J. Huang and C. X. Ling, "Using AUC and accuracy in evaluating learning algorithms," IEEE Trans. Knowl. Data Eng., vol. 17, no. 3, pp. 299-310, Mar. 2005.
16. J. Alcalá-Fdez, A. Fernández, J. Luengo, J. Derrac, S. García, L. Sánchez, and F. Herrera, "KEEL data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework," J. Multiple-Valued Logic Soft Comput., vol. 17, nos. 2-3, pp. 255-287, 2010.
17. T. Cover and P. Hart, "Nearest neighbor pattern classification, " IEEE Trans. Inf. Theory, vol. 13, no. 1, pp. 21-27, Jan. 1967.
18. J. R. Quinlan, C4.5: Programs for Machine Learning. San Francisco, CA: Morgan Kaufmann, 1993.
19. J. Hühn and E. Hüllermeier, "FURIA: An algorithm for unordered fuzzy rule induction," Data Mining Knowl. Discovery, vol. 19, no. 3, pp. 293-319, 2009.
20. R. Fisher, "The use of multiple measurements in taxonomic problems," Ann. Eugen., vol. 7, no. 2, pp. 179-188, 1936.
21. E. Frank and I. Witten, "Generating accurate rule sets without global optimization," in Proc. 15th Int. Conf. Mach. Learn., 1998, pp. 144-151.
22. Y. Chen and J. Wang, "Support vector learning for fuzzy rule-based classification systems," IEEE Trans. Fuzzy Syst., vol. 11, no. 6, pp. 716-728, Dec. 2003.
23. W. Cohen, "Fast effective rule induction," in Proc. 12th Int. Conf. Mach. Learn., 1995, pp. 1-10.
24. W. C. Cohen, "Fast effective rule induction," in Proc. 12th Int. Conf. Mach. Learn., 1995, pp. 115-123.
25. J. Alcalá-Fdez, L. Sánchez, S. García, M. J. D. Jesus, S. Ventura, J. M. Garrell, J. Otero, J. Bacardit, V. M. Rivas, J. C. Fernández, and F. Herrera, "Keel: A software tool to assess evolutionary algorithms for data mining problems," Soft Comput. Fusion Found., Methodol. Appl., vol. 13, no. 3, pp. 307-318, 2009.
26. S. García, A. Fernández, J. Luengo, and F. Herrera, "Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power," Inf. Sci., vol. 180, no. 10, pp. 2044-2064, 2010.
27. T. Raeder, T. R. Hoens, and N. V. Chawla, "Consequences of variability in classifier performance estimates," in Proc. IEEE Int. Conf. Data Mining, Dec. 2010, pp. 421-430.