Performance-Estimation Properties of Cross-Validation-Based Protocols with Simultaneous Hyper-Parameter Optimization

International Journal on Artificial Intelligence Tools

Volumn 24, Issue 5, 2015, Pages

  Tsamardinos, Ioannis ^a,b   Rakhshani, Amin ^a,b   Lagani, Vincenzo ^b  

^a UNIVERSITY OF CRETE   (Greece)

^b INSTITUTE OF COMPUTER SCIENCE   (Greece)

Author keywords

comparative evaluation; cross validation; model selection; Performance estimation; stratification

Indexed keywords

LEARNING SYSTEMS; THERMAL STRATIFICATION;

COMPARATIVE EVALUATIONS; COMPUTATIONAL EXPERIMENT; CROSS VALIDATION; HYPER-PARAMETER OPTIMIZATIONS; MODEL SELECTION; NESTED CROSS VALIDATIONS; PERFORMANCE ESTIMATION; STATISTICAL COMPARISONS;

PARAMETER ESTIMATION;

EID: 84944598781     PISSN: 02182130     EISSN: 17936349     Source Type: Journal    
DOI: 10.1142/S0218213015400230     Document Type: Conference Paper

References (36)

1. D. Anguita, A. Ghio, L. Oneto and S. Ridella, In-sample and out-of-sample model selection and error estimation for support vector machines, IEEE Trans. Neural Networks Learn. Syst. 23(9) (September 2012) 1390-1406.
2. G. C. Cawley and N. L. C. Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, J. Mach. Learn. Res. 11 (March 2010) 2079-2107.
3. D. D. Jensen and P. R. Cohen, Multiple comparisons in induction algorithms, Mach. Learn. 38 (2000) 309-338.
4. R. J. Tibshirani and R. Tibshirani, A bias correction for the minimum error rate in crossvalidation, Ann. Appl. Stat. 3(2) (June 2009) 822-829.
5. R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in Int. Joint Conf. on Artificial Intelligence (1995), Vol. 14, pp. 1137-1143.
6. A. Statnikov, C. F. Aliferis, I. Tsamardinos, D. Hardin and S. Levy, A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis, Bioinformatics 21(5) (March 2005) 631-643.
7. R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification (October 2000), 2nd edn.
8. T. M. Mitchell, Machine Learning (March 1997).
9. C. M. Bishop, Pattern Recognition and Machine Learning, Information Science and Statistics) (August 2006).
10. T. Hastie, R. Tibshirani and J. Friedman, The elements of statistical learning, Elements 1 (2009) 337-387.
11. Y. Bengio and Y. Grandvalet, Bias in estimating the variance of K-fold cross-validation, in Statistical Modeling and Analysis for Complex Data Problem 1 (2005) 75-95.
12. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann Series in Data Management Systems (June 2005), 2nd edn.
13. V. Lagani and I. Tsamardinos, Structure-based variable selection for survival data, Bioinformatics 26(15) (2010) 1887-1894.
14. A. Statnikov, I. Tsamardinos, Y. Dosbayev and C. F. Aliferis, GEMS: A system for automated cancer diagnosis and biomarker discovery from microarray gene expression data, Int. J. Med. Inform. 74(7-8) (August 2005) 491-503.
15. S. Salzberg, On comparing classifiers: Pitfalls to avoid and a recommended approach, Data Min. Knowl. Discov. 328 (1997) 317-328.
16. N. Iizuka, M. Oka, H. Yamada-Okabe, M. Nishida, Y. Maeda, N. Mori, T. Takao, T. Tamesa, A. Tangoku, H. Tabuchi, K. Hamada, H. Nakayama, H. Ishitsuka, T. Miyamoto, A. Hirabayashi, S. Uchimura and Y. Hamamoto, Oligonucleotide microarray for prediction of early intrahepatic recurrence of hepatocellular carcinoma after curative resection, Lancet 361(9361) (March 2003) 923-929.
17. L. A. Kurgan, K. J. Cios, R. Tadeusiewicz, M. Ogiela and L. S. Goodenday, Knowledge discovery approach to automated cardiac SPECT diagnosis, Artif. Intell. Med. 23(2) (October 2001) 149-169.
18. R. K. Bock, A. Chilingarian, M. Gaug, F. Hakl, T. Hengstebeck, M. Jiina, J. Klaschka, E. Kotr, P. Savický, S. Towers, A. Vaiciulis and W. Wittek, Methods for multidimensional event classification: A case study using images from a Cherenkov gamma-ray telescope, Nucl. Instruments Methods Phys. Res. Sect. A: Accel. Spectrometers, Detect. Assoc. Equip. 516(2-3) (January 2004) 511-528.
19. K. Mansouri, T. Ringsted, D. Ballabio, R. Todeschini and V. Consonni, Quantitative structureactivity relationship models for ready biodegradability of chemicals, J. Chem. Inf. Model 53 (2013) 867-878.
20. S. Moro and R. M. S. Laureano, Using data mining for bank direct marketing: An application of the CRISP-DM methodology, Eur. Simul. Model. Conf. (2011), pp. 117-121.
21. S. C. Bendall, E. F. Simonds, P. Qiu, E. D. Amir, P. O. Krutzik, R. Finck, R. V Bruggner, R. Melamed, A. Trejo, O. I. Ornatsky, R. S. Balderas, S. K. Plevritis, K. Sachs, D. Pe'er, S. D. Tanner and G. P. Nolan, Single-cell mass cytometry of differential immune and drug responses across a human hematopoietic continuum, Science 332(6030) (May 2011) 687-696.
22. M. Sikora and L. Wrobel, Application of rule induction algorithms for analysis of data collected by seismic hazard monitoring systems in coal mines, Arch. Min. Sci. 55(1) (2010) 91-114.
23. A. A. Aguilar-Arevalo, A. O. Bazarko, S. J. Brice, B. C. Brown, L. Bugel, J. Cao, L. Coney, J. M. Conrad, D. C. Cox, A. Curioni, Z. Djurcic, D. A. Finley, B. T. Fleming, R. Ford, F. G. Garcia, G. T. Garvey, C. Green, J. A. Green, T. L. Hart, E. Hawker, R. Imlay, R. A. Johnson, P. Kasper, T. Katori, T. Kobilarcik, I. Kourbanis, S. Koutsoliotas, E. M. Laird, J. M. Link, Y. Liu, Y. Liu, W. C. Louis, K. B. M. Mahn, W. Marsh, P. S. Martin, G. McGregor, W. Metcalf, P. D. Meyers, F. Mills, G. B. Mills, J. Monroe, C. D. Moore, R. H. Nelson, P. Nienaber, S. Ouedraogo, R. B. Patterson, D. Perevalov, C. C. Polly, E. Prebys, J. L. Raaf, H. Ray, B. P. Roe, A. D. Russell, V. Sandberg, R. Schirato, D. Schmitz, M. H. Shaevitz, F. C. Shoemaker, D. Smith, M. Sorel, P. Spentzouris, I. Stancu, R. J. Stefanski, M. Sung, H. A. Tanaka, R. Tayloe, M. Tzanov, R. Van de Water, M. O. Wascko, D. H. White, M. J. Wilking, H. J. Yang, G. P. Zeller and E. D. Zimmerman, Search for electron neutrino appearance at the m2 ~ 1 eV2 scale, Phys. Rev. Lett. 98 (2007) 231801.
24. D. Coppersmith, S. J. Hong and J. R. M. Hosking, Partitioning nominal attributes in decision trees, Data Min. Knowl. Discov. 3 (1999) 197-217.
25. C.-C. Chang and C.-J. Lin, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol. 2(27) (2011) 1-27.
26. I. Tsamardinos, V. Lagani and D. Pappas, Discovering multiple, equivalent biomarker signatures, in 7th Conference of the Hellenic Society for Computational Biology and Bioinformatics (HSCBB12) (2012).
27. I. Tsamardinos, L. E. Brown and C. F. Aliferis, The max-min hill-climbing Bayesian network structure learning algorithm, Mach. Learn. 65(1) (2006) 31-78.
28. T. Fawcett, An introduction to ROC analysis, Pattern Recognit. Lett. 27 (2006) 861-874.
29. A. Airola, T. Pahikkala, W. Waegeman, B. De Baets and T. Salakoski, A comparison of AUC estimators in small-sample studies, J. Mach. Learn. Res. W&CP 8 (2010) 3-13.
30. R. G. O'brien, A general ANOVA method for robust tests of additive models for variances, J. Am. Stat. Assoc. 74(368) (December 1979) 877-880.
31. M. H. Quenouille, Approximate tests of correlation in time-series 3, Math. Proc. Cambridge Philos. Soc. 45(03) (October 1949) 483-484.
32. S. Varma and R. Simon, Bias in error estimation when using cross-validation for model selection, BMC Bioinformatics 7 (January 2006) 91.
33. I. Tsamardinos, V. Lagani and A. Rakhshani, Performance-estimation properties of crossvalidation-based protocols with simultaneous hyper-parameter optimization, in Proc. of the 8th Hellenic Conference on Artificial Intelligence (SETN'14) (2014).
34. C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning (2006), Vol. 14.
35. T. Hofmann, B. Schölkopf and A. J. Smola, Kernel methods in machine learning, The Annals of Statistics 36(3) (2008) 1171-1220.
36. D. Duvenaud, J. Lloyd, R. Grosse, J. Tenenbaum and Z. Ghahramani, Structure discovery in nonparametric regression through compositional kernel search, in Proc. of the Int. Conf. on Machine Learning (ICML) (2013), Vol. 30, pp. 1166-1174.