Overfitting in linear feature extraction for classification of high-dimensional image data

Pattern Recognition

Volumn 53, Issue , 2016, Pages 73-86

  Liu, Raymond ^a   Gillies, Duncan F ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Classification; Dimensionality reduction; Feature extraction; High dimensional datasets; Overfitting

Indexed keywords

EXTRACTION; FEATURE EXTRACTION; IMAGE CLASSIFICATION;

CLASSIFICATION PERFORMANCE; DIMENSIONALITY REDUCTION; DIMENSIONALITY REDUCTION TECHNIQUES; HIGH DIMENSIONAL DATASETS; HIGH-DIMENSIONAL IMAGES; LINEAR FEATURE EXTRACTION; OVERFITTING; SUBSPACE PROJECTION;

CLASSIFICATION (OF INFORMATION);

EID: 84958673886     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2015.11.015     Document Type: Article

References (41)

1. A. Jain, R. Duin, and J. Mao Statistical pattern recognition: a review IEEE Trans. Pattern Anal. Mach. Intell. 22 1 2000 4 37
2. C.M. Bishop Pattern Recognition and Machine Learning 2006 Springer-Verlag New York, Inc., Secaucus, NJ, USA
3. A. Jain Dimensionality and sample size considerations in pattern recognition practice Handbook of Statistics, Krishnaiah 2 1982 835 855
4. J.H. Friedman On bias, variance, 0/1loss, and the curse-of-dimensionality Data Min. Knowl. Discov. 1 1 1997 55 77
5. S.J. Raudys, and A.K. Jain Small sample size effects in statistical pattern recognition: recommendations for practitioners IEEE Trans. Pattern Anal. Mach. Intell. 13 3 1991 252 264
6. Huang, R., Liu, Q., Lu, H., Ma, S. (2002). Solving the small sample size problem of LDA. In Pattern Recognition, 2002. Proceedings. 16th International Conference on (Vol. 3, pp. 29-32). IEEE, Quebec City, Quebec, Canada.
7. G. Trunk A problem of dimensionality: a simple example IEEE Trans. Pattern Anal. Mach. Intell. 3 1979 306 307
8. M.A. Babyak What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models Psychosom. Med. 66 3 2004 411 421
9. D.M. Hawkins The problem of overfitting J. Chem. Inf. Comput. Sci. 44 1 2004 1 12
10. G.C. Cawley, and N.L. Talbot Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters J. Mach. Learn. Res. 8 2007 841 861
11. G.C. Cawley, and N.L. Talbot On over-fitting in model selection and subsequent selection bias in performance evaluation J. Mach. Learn. Res. 11 2010 2079 2107
12. Lawrence, S., Giles, C. L. (2000). Overfitting and neural networks: conjugate gradient and backpropagation. In Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on (Vol. 1, pp. 114-119). IEEE, Como, Italy.
13. M. Sugiyama, T. Idé, S. Nakajima, and J. Sese Semi-supervised local fisher discriminant analysis for dimensionality reduction Mach. Learn. 78 1-2 2010 35 61
14. R. Liu, and D.F. Gillies An estimate of mutual information that permits closed-form optimisation Entropy 15 5 2013 1690 1704 10.3390/e15051690
15. W.T. Freeman, and E.H. Adelson The design and use of steerable filters IEEE Trans. Pattern Anal. Mach. Intell. 9 1991 891 906
16. M. Jacob, and M. Unser Design of steerable filters for feature detection using canny-like criteria IEEE Trans. Pattern Anal. Mach. Intell. 26 8 2004 1007 1019
17. S.-H. Jeng, H.-Y. M. Liao, Y.-T. Liu, M.-Y. Chern, An efficient approach for facial feature detection using geometrical face model, in: Proceedings of the 13th International Conference on Pattern Recognition, vol. 3, Vienna, Austria: IEEE, 1996, pp. 426-430.
18. I. Guyon, and A. Elisseeff An introduction to variable and feature selection J. Mach. Learn. Res. 3 2003 1157 1182
19. Y. Saeys, I. Inza, and P. Larrañaga A review of feature selection techniques in bioinformatics Bioinformatics 23 19 2007 2507 2517
20. D.K. Slonim, P. Tamayo, J.P. Mesirov, T.R. Golub, E.S. Lander, Class prediction and discovery using gene expression data, in: Proceedings of the Fourth Annual International Conference on Computational Molecular Biology, Tokyo, Japan: ACM, 2000, pp. 263-272.
21. C. Ding, and H. Peng Minimum redundancy feature selection from microarray gene expression data J. Bioinform. Comput. Biol. 3 02 2005 185 205
22. H. Peng, F. Long, and C. Ding Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy IEEE Trans. Pattern Anal. Mach. Intell. 27 2005 1226 1238 (http://dx.doi.org/http://doi.ieeecomputersociety.org/10.1109/TPAMI.2005.159)
23. I.S. Dhillon, S. Mallela, and R. Kumar A divisive information theoretic feature clustering algorithm for text classification J. Mach. Learn. Res. 3 2003 1265 1287
24. G. Forman An extensive empirical study of feature selection metrics for text classification J. Mach. Learn. Res. 3 2003 1289 1305
25. J.B. Tenenbaum, V. De Silva, and J.C. Langford A global geometric framework for nonlinear dimensionality reduction Science 290 5500 2000 2319 2323
26. J. Leiva-Murillo, and A. Artes-Rodriguez Maximization of mutual information for supervised linear feature extraction IEEE Trans. Neural Netw. 18 5 2007 1433 1441
27. K. Torkkola Feature extraction by non parametric mutual information maximization J. Mach. Learn. Res. 3 2003 1415 1438
28. J. Kapur Measures of Information and their Applications 1994 Wiley New Delhi, India
29. J. Yang, and J.-y. Yang Why can LDA be performed in PCA transformed space? Pattern Recognit. 36 2 2003 563 566
30. X. Qiu, and L. Wu Info-margin maximization for feature extraction Pattern Recognit. Lett. 30 16 2009 1516 1522
31. H. Li, T. Jiang, and K. Zhang Efficient and robust feature extraction by maximum margin criterion IEEE Trans. Neural Netw. 17 1 2006 157 165
32. L. Van der Maaten, E. Postma, and H. Van den Herik Dimensionality reduction: a comparative review J. Mach. Learn. Res. 10 2009 1 41
33. B. Schölkopf, A. Smola, and K.-R. Müller Nonlinear component analysis as a kernel eigenvalue problem Neural Comput. 10 5 1998 1299 1319
34. K. Fukunaga Introduction to Statistical Pattern Recognition 1990 Academic Press San Diego, CA, USA
35. T. Zhang, D. Tao, X. Li, and J. Yang Patch alignment for dimensionality reduction IEEE Trans. Knowl. Data Eng. 21 9 2009 1299 1313
36. M. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, Coding facial expressions with gabor wavelets, in: Proceedings of Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan: IEEE, 1998, pp. 200-205.
37. The Essex Face Database (http://cswww.essex.ac.uk/mv/allfaces/) (accessed: 11.06.13).
38. Fingerprint Verification Competition Database (http://bias.csr.unibo.it/fvc2000/download.asp) (accessed: 12.06.13).
39. Sheffield Face Database (http://www.sheffield.ac.uk/eee/research/iel/research/face) (accessed: 12.06.13).
40. T. Mitchell Machine Learning 1997 McGraw-Hill Boston, MA, USA
41. S.S. Rao Engineering Optimization, Theory and Practice 2009 Wiley Hoboken, NJ, USA