Density-preserving sampling: Robust and efficient alternative to cross-validation for error estimation

IEEE Transactions on Neural Networks and Learning Systems

Volumn 24, Issue 1, 2013, Pages 22-34

  Budka, Marcin ^a   Gabrys, Bogdan ^a  

^a BOURNEMOUTH UNIVERSITY   (United Kingdom)

Author keywords

Bootstrap; Correntropy; Cross validation; Error estimation; Model selection; Sampling

Indexed keywords

BOOTSTRAP; CORRENTROPY; CROSS VALIDATION; ESTIMATION PROCEDURES; GENERALIZATION ABILITY; GENERALIZATION ERROR ESTIMATIONS; MODEL SELECTION; RANKING AND SELECTION;

BENCHMARKING; ERROR ANALYSIS; REGRESSION ANALYSIS; SAMPLING;

ESTIMATION;

EID: 84884946129     PISSN: 2162237X     EISSN: 21622388     Source Type: Journal    
DOI: 10.1109/TNNLS.2012.2222925     Document Type: Article

References (37)

1. R. Duda, P. Hart, and D. Stork, Pattern Classification, 2nd ed. New York: Wiley, 2001.
2. T. Cover, "Learning in pattern recognition," Stanford Electronics Laboratories, Stanford Univ., Stanford, CA, Tech. Rep. TR-6151-5, 1968.
3. B. Efron, "Bootstrap methods: Another look at the jackknife," Ann. Stat., vol. 7, no. 1, pp. 1-26, 1979.
4. R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," in Proc. 14th Int. Joint Conf. Artif. Intell., vol. 2. 1995, pp. 1137-1145.
5. A. Harzing. (2010). Publish or Perish [Online]. Available: http://www.harzing.com/pop.htm
6. W. Liu, P. Pokharel, and J. Principe, "Correntropy: A localized similarity measure," in Proc. Int. Joint Conf. Neural Netw., 2006, pp. 4919-4924.
7. M. Budka and B. Gabrys, "Correntropy-based density-preserving data sampling as an alternative to standard cross-validation," in Proc. IEEE World Congr. Comput. Intell., Oct. 2010, pp. 1437-1444.
8. D. Anguita, A. Ghio, L. Oneto, and S. Ridella, "In-sample and out-of-sample model selection and error estimation for support vector machines," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 9, pp. 1390-1406, Sep. 2012.
9. A. Antos, L. Devroye, and L. Gyorfi, "Lower bounds for Bayes error estimation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 7, pp. 643-645, Jul. 1999.
10. A. Jain, R. Duin, and J. Mao, "Statistical pattern recognition: A review," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 4-37, Jan. 2000.
11. S. Weiss and C. Kulikowski, Computer Systems that Learn. San Francisco, CA: Morgan Kaufmann, 1991.
12. B. Efron and R. Tibshirani, An Introduction to the Bootstrap, vol. 57. New York: Chapman & Hall, 1993.
13. A. Molinaro, R. Simon, and R. Pfeiffer, "Prediction error estimation: A comparison of resampling methods," Bioinformatics, vol. 21, no. 15, pp. 3301-3307, 2005. (Pubitemid 41418445)
14. J. Kim, "Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap," Comput. Stat. Data Anal., vol. 53, no. 11, pp. 3735-3745, 2009.
15. S. Borra and A. Di Ciaccio, "Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods," Comput. Stat. Data Anal., vol. 54, no. 12, pp. 2976-2989, 2010.
16. Y. Dodge, D. Cox, D. Commenges, A. Davison, and P. Solomon, The Oxford Dictionary of Statistical Terms. Oxford, U.K.: Oxford Univ. Press, 2006.
17. A. Cichocki and S. Amari, "Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities," Entropy, vol. 12, no. 6, pp. 1532-1568, 2010.
18. S. Kullback and R. Leibler, "On information and sufficiency," Ann. Math. Stat., vol. 22, no. 1, pp. 79-86, 1951.
19. J. Lin, "Divergence measures based on the Shannon entropy," IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 145-151, Jan. 1991.
20. M. Budka, B. Gabrys, and K. Musial, "On accuracy of PDF divergence estimators and their applicability to representative data sampling," Entropy, vol. 13, no. 7, pp. 1229-1266, 2011.
21. M. Budka, "Physically inspired methods and development of data-driven predictive systems," Ph.D. dissertation, Dept. Electr. Eng., Bournemouth Univ., Dorset, U.K., 2010.
22. J. Principe, D. Xu, and J. Fisher, "Information theoretic learning," in Unsupervised Adaptive Filtering, S. Haykin, Ed. New York: Wiley, 2000, pp. 265-319.
23. J. Principe, D. Xu, Q. Zhao, and J. Fisher, "Learning from examples with information theoretic criteria," J. VLSI Signal Process., vol. 26, no. 1, pp. 61-77, 2000.
24. I. Santamaría, P. Pokharel, and J. Principe, "Generalized correlation function: Definition, properties, and application to blind equalization," IEEE Trans. Signal Process., vol. 54, no. 6, pp. 2187-2197, Jun. 2006. (Pubitemid 43811412)
25. I. Santamaría, P. Pokharel, and J. Principe, "Generalized correlation function: Definition, properties, and application to blind equalization," IEEE Trans. Signal Process., vol. 54, no. 6, pp. 2187-2197, Jun. 2006. (Pubitemid 43811412)
26. W. Liu, P. Pokharel, and J. Principe, "Correntropy: Properties and applications in non-Gaussian signal processing," IEEE Trans. Signal Process., vol. 55, no. 11, pp. 5286-5298, Nov. 2007. (Pubitemid 350130741)
27. W. Liu, P. Pokharel, and J. Principe, "Error entropy, correntropy and M-estimation," in Proc. 16th IEEE Signal Process. Soc. Workshop Mach. Learn. Signal Process., Nov. 2006, pp. 179-184.
28. M. Budka and B. Gabrys, "Ridge regression ensemble for toxicity prediction," Procedia Comput. Sci., vol. 1, no. 1, pp. 193-201, 2010.
29. J. Bentley, "Multidimensional binary search trees used for associative searching," Commun. ACM, vol. 18, no. 9, pp. 509-517, 1975.
30. A. Asuncion and D. Newman. (2007). UCI Machine Learning Repository [Online]. Available: http://www.ics.uci.edu/~mlearn/MLRepository.html
31. Enhanced Learning for Evolutive Neural Architecture (ELENA) Database. (1995) [Online]. Available: http://www.dice.ucl.ac.be/mlg/?page=Elena
32. R. Duin, P. Juszczak, P. Paclik, E. Pekalska, D. de Ridder, D. Tax, and S. Verzakov. (2007). PR-Tools 4.1: A MATLAB Toolbox for Pattern Recognition [Online]. Available: http://prtools.org
33. L. Kuncheva, Fuzzy Classifier Design. Heidelberg, Germany: Physica Verlag, 2000.
34. B. Ripley, Pattern Recognition and Neural Networks. Cambridge, U.K.: Cambridge Univ. Press, 1996.
35. L. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms. New York: Wiley, 2004.
36. X. Zeng and T. Martinez, "Distribution-balanced stratified cross-validation for accuracy estimation," J. Experim. Theoret. Artif. Intell., vol. 12, no. 1, pp. 1-12, 2000.
37. J. Moreno-Torres, J. Sáez, and F. Herrera, "Study on the impact of partition-induced dataset shift on k-fold cross-validation," IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 8, pp. 1304-1312, Aug. 2012.