Out-of-sample error estimation: The blessing of high dimensionality

IEEE International Conference on Data Mining Workshops, ICDMW

Volumn 2015-January, Issue January, 2015, Pages 637-644

  Oneto, Luca ^a   Ghio, Alessandro ^a   Ridella, Sandro ^a   Ortiz, Jorge Luis Reyes ^b   Anguita, Davide ^b  

^a UNIVERSITY OF GENOA   (Italy)

^b UNIVERSITY OF GENOA   (Italy)

Author keywords

Classification; Error Estimation; High Dimensional Problems; Performance Estimation; Small Sample; Supervised Learning

Indexed keywords

DATA MINING; ERROR ANALYSIS; ERRORS; GENE EXPRESSION; SUPERVISED LEARNING;

ARTIFICIAL PROBLEMS; HIGH DIMENSIONALITY; HIGH-DIMENSIONAL PROBLEMS; OUT-OF-SAMPLE ERRORS; PERFORMANCE ESTIMATION; SMALL SAMPLES; STATISTICAL INFERENCE; SUPERVISED CLASSIFICATION;

CLASSIFICATION (OF INFORMATION);

EID: 84936875546     PISSN: 23759232     EISSN: 23759259     Source Type: Conference Proceeding    
DOI: 10.1109/ICDMW.2014.41     Document Type: Conference Paper

References (45)

1. D. L. Donoho, "High-dimensional data analysis: The curses and blessings of dimensionality, " AMS Math Challenges Lecture, pp. 1-32, 2000.
2. D. E. Bassett, M. B. Eisen, and M. S. Boguski, "Gene expression informatics-it's all in your mine, " Nature genetics, vol. 21, pp. 51-55, 1999.
3. A. Webb and G. C. Kagadis, "Introduction to biomedical imaging, " Medical Physics, vol. 30, no. 8, pp. 2267-2267, 2003.
4. N. M. Ball, R. J. Brunner, A. D. Myers, N. E. Strand, S. L. Alberts, and D. Tcheng, "Robust machine learning applied to astronomical data sets probabilistic photometric redshifts for galaxies and quasars in the sdss and galex, " The Astrophysical Journal, vol. 683, no. 1, pp. 12-21, 2008.
5. C. Ambroise and G. J. McLachlan, "Selection bias in gene extraction on the basis of microarray gene-expression data, " Proceedings of the national academy of sciences, vol. 99, no. 10, pp. 6562-6566, 2002.
6. M. Verleysen, "Learning high-dimensional data, " Nato Science Series Sub Series III Computer And Systems Sciences, vol. 186, pp. 141-162, 2003.
7. S. Saremi and T. J. Sejnowski, "On criticality in highdimensional data, " Neural computation, vol. 26, no. 7, pp. 1329-1339, 2014.
8. U. M. Braga-Neto and E. R. Dougherty, "Is cross-validation valid for small-sample microarray classification" Bioinformatics, vol. 20, no. 3, pp. 374-380, 2004.
9. D. Anguita, A. Ghio, L. Oneto, and S. Ridella, "In-sample and out-of-sample model selection and error estimation for support vector machines, " IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no. 9, pp. 1390-1406, 2012.
10. M. Theus, "High-dimensional data visualization, " in Handbook of data visualization, 2008.
11. A. Ultsch, "Maps for the visualization of high-dimensional data spaces, " in Workshop on Self-Organizing Maps, 2003.
12. M. E. Houle, "Dimensionality, discriminability, density and distance distributions, " in IEEE International Conference on Data Mining Workshops (International Workshop on High Dimensional Data Mining), 2013.
13. A. Flexer and D. Schnitzer, "Can shared nearest neighbors reduce hubness in high-dimensional spaces" in IEEE International Conference on Data Mining Workshops (International Workshop on High Dimensional Data Mining), 2013.
14. A. Kabán, "Non-parametric detection of meaningless distances in high dimensional data, " Statistics and Computing, vol. 22, no. 2, pp. 375-385, 2012.
15. P. Jiang and M. T. Heath, "Pattern discovery in high dimensional binary data, " in IEEE International Conference on Data Mining Workshops (International Workshop on High Dimensional Data Mining), 2013.
16. H. Zou and T. Hastie, "Regularization and variable selection via the elastic net, " Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 67, no. 2, pp. 301-320, 2005.
17. M. Belkin and P. Niyogi, "Towards a theoretical foundation for laplacian-based manifold methods, " in Learning theory, 2005.
18. F. Petitjean, G. I. Webb, and A. E. Nicholson, "Scaling loglinear analysis to high-dimensional data, " in IEEE International Conference on Data Mining, 2013.
19. Y. Wang, X. Chang, R. Li, and Z. Xu, "Sparse k-means with the lq (0 i q i 1) constraint for high-dimensional data clustering, " in IEEE International Conference on Data Mining, 2013.
20. A. Kabán, "A new look at compressed ordinary least squares, " in IEEE International Conference on Data Mining Workshops (International Workshop on High Dimensional Data Mining), 2013.
21. J. Anderson, M. Belkin, N. Goyal, R. L., and J. Voss, "The more, the merrier: the blessing of dimensionality for learning large Gaussian mixtures, " Journal of Machine Learning Research, vol. 35, pp. 1-30, 2014.
22. H. White, "A reality check for data snooping, " Econometrica, vol. 68, no. 5, pp. 1097-1126, 2000.
23. D. Jensen, "Data snooping, dredging and fishing: The dark side of data mining a sigkdd99 panel report, " ACM SIGKDD Explorations Newsletter, vol. 1, no. 2, pp. 52-54, 2000.
24. P. L. Bartlett, S. Boucheron, and G. Lugosi, "Model selection and error estimation, " Machine Learning, vol. 48, no. 1-3, pp. 85-113, 2002.
25. L. Roberge, S. Long, and D. Burnham, "Data warehouses and data mining tools for the legal profession: using information technology to raise the standard of practice, " Syracuse Law Review, vol. 52, pp. 1281-1292, 2002.
26. V. N. Vapnik, Statistical learning theory. New York Wiley, 1998.
27. P. L. Bartlett and S. Mendelson, "Rademacher and Gaussian complexities: Risk bounds and structural results, " The Journal of Machine Learning Research, vol. 3, pp. 463-482, 2003.
28. O. Bousquet and A. Elisseeff, "Stability and generalization, " The Journal of Machine Learning Research, vol. 2, pp. 499-526, 2002.
29. E. Parrado-Hernández, A. Ambroladze, J. Shawe-Taylor, and S. Sun, "Pac-bayes bounds with data dependent priors, " The Journal of Machine Learning Research, vol. 13, no. 1, pp. 3507-3531, 2012.
30. J. Langford, "Tutorial on practical prediction theory for classification, " Journal of machine learning research, vol. 6, no. 1, p. 273, 2006.
31. V. N. Vapnik, "An overview of statistical learning theory, " IEEE Transactions on Neural Networks, vol. 10, no. 5, pp. 988-999, 1999.
32. J. Han, M. Kamber, and J. Pei, Data mining: concepts and techniques. Morgan kaufmann, 2006.
33. C. J. Clopper and E. S. Pearson, "The use of confidence or fiducial limits illustrated in the case of the binomial, " Biometrika, vol. 26, no. 4, pp. 404-413, 1934.
34. D. Anguita, L. Ghelardoni, A. Ghio, and S. Ridella, "Test error bounds for classifiers: A survey of old and new results, " in IEEE Symposium on Foundations of Computational Intelligence, 2011.
35. B. Efron, The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
36. S. Arlot and A. Celisse, "A survey of cross-validation procedures for model selection, " Statistics Surveys, vol. 4, pp. 40-79, 2010.
37. L. K. Hansen and P. Salamon, "Neural network ensembles, " IEEE transactions on pattern analysis and machine intelligence, vol. 12, no. 10, pp. 993-1001, 1990.
38. A. Statnikov, M. Henaff, V. Narendra, K. Konganti, Z. Li, L. Yang, Z. Pei, M. J. Blaser, C. F. Aliferis, and A. V. Alekseyenko, "A comprehensive evaluation of multicategory classification methods for microbiomic data, " Microbiome, vol. 1, no. 1, p. 11, 2013.
39. A. Statnikov, I. Tsamardinos, and Y. Dosbayev, "GEMS: a system for automated cancer diagnosis and biomarker discovery from microarray gene expression data, " International Journal of Medical Informatics, vol. 74, no. 7-8, pp. 491-503, 2005.
40. N. Ancona, R. Maglietta, A. Piepoli, A. D'Addabbo, R. Cotugno, M. Savino, S. Liuni, M. Carella, G. Pesole, and F. Perri, "On the statistical assessment of classifiers using DNA microarray data, " Bioinformatics, vol. 7, no. 1, pp. 387-399, 2006.
41. U. Alon, N. Barkai, D. A. Notterman, K. Gish, S. Ybarra, D. Mack, and A. Levine, "Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays, " National Academy of Sciences of the United States of America, vol. 96, no. 12, pp. 6745-6767, 1999.
42. M. West, C. Blanchette, H. Dressman, E. Huang, S. Ishida, R. Spang, H. Zuzan, J. Olson, J. Marks, and J. Nevins, "Predicting the clinical status of human breast cancer by using gene expression profiles, " National Academy of Sciences of the United States of America, vol. 98, no. 20, pp. 11 462-11 490, 2001.
43. T. R. Golub, D. K. Slonim, P. Tamayo, C. Huard, M. Gaasenbeek, J. P. Mesirov, H. Coller, M. L. Loh, J. R. Downing, M. A. Caligiuri, C. D. Bloomfield, and E. S. Lander, "Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring, " Science, vol. 286, no. 5439, pp. 531-537, 1999.
44. D. Page, F. Zhan, J. Cussens, M. Waddell, J. Hardin, B. Barlogie, and J. Shaughnessy Jr, "Comparative data mining for microarrays: A case study based on multiple myeloma, " in International Conference on Intelligent Systems for Molecular Biology, 2002.
45. T. K. Ho, "The random subspace method for constructing decision forests, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832-844, 1998.