An efficient discriminant-based solution for small sample size problem

Pattern Recognition

Volumn 42, Issue 5, 2009, Pages 857-866

  Das, Koel ^a   Nenadic, Zoran ^a  

^a University of California at Irvine   (United States)

Author keywords

Bayes error; Classification; Feature extraction; Linear discriminant analysis; Principal component analysis

Indexed keywords

BIOINFORMATICS; COMPUTER AIDED DIAGNOSIS; DISCRIMINANT ANALYSIS; FACE RECOGNITION; FEATURE EXTRACTION; IMAGE RETRIEVAL; PATTERN RECOGNITION; PIECEWISE LINEAR TECHNIQUES;

BAYES ERROR; BRAIN IMAGING; CLASSIFICATION; CLASSIFICATION RATES; DATA SETS; DIGIT RECOGNITIONS; DIMENSIONAL DATUMS; LINEAR DISCRIMINANT ANALYSIS; LINEAR DISCRIMINANTS; LINEAR SUBSPACES; PIECEWISE LINEAR; SMALL SAMPLE SIZES; STANDARD PATTERNS; STATISTICAL DATUMS;

PRINCIPAL COMPONENT ANALYSIS;

EID: 58249088710     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2008.08.036     Document Type: Article

References (78)

1. Duda R.O., Hart P.E., and Stork D.G. Pattern Classification (2001), Wiley-Interscience, New York
2. Fisher R.A. The use of multiple measurements in taxonomic problems. Ann. Eugenics 7 (1936) 179-188
3. Rao C.R. The utilization of multiple measurements in problems of biological classification. J. R. Stat. Soc. B Methods 10 2 (1948) 159-203
4. Lewis II P.M. The characteristic selection problem in recognition systems. IEEE Trans. Inf. Theory 8 2 (1962) 171-178
5. H.P. Decell, J.A. Quirein, An iterative approach to the feature selection problem, in: Proceedings of Purdue Conference on Machine Processing of Remotely Sensed Data, 1972, pp. 3B1-3B12.
6. Devijver P.A., and Kittler J. Pattern Recognition: A Statistical Approach (1982), Prentice-Hall International, Englewood Cliffs, NJ
7. Kumar N., and Andreou A.G. Heteroscedastic discriminant analysis and reduced rank hmms for improved speech recognition. Speech Commun. 26 4 (1998) 283-297
8. G. Saon, M. Padmanabhan, Minimum Bayes error feature selection for continuous speech recognition, in: Advances in Neural Information Processing Systems, vol. 13, MIT Press, Cambridge, MA, 2001, pp. 800-806.
9. Torkkola K. Discriminative features for document classification. Proceedings 16th International Conference on Pattern Recognition vol. 1 (2002) 472-475
10. Loog M., and Duin R.P.W. Linear dimensionality reduction via a heteroscedastic extension of LDA: the Chernoff criterion. IEEE Trans. Pattern Anal. 26 (2004) 732-739
11. Nenadic Z. Information discriminant analysis: feature extraction with and information theoretic objective. IEEE Trans. Pattern Anal. 29 8 (2007) 1394-1407
12. Das K., and Nenadic Z. Approximate information discriminant analysis: a computationally simple heteroscedastic feature extraction technique. Pattern Recognition 41 5 (2008) 1565-1574
13. Fukunaga K. Introduction to Statistical Pattern Recognition. second ed. (1990), Academic Press, New York
14. Turk M., and Pentland A. Eigenfaces for recognition. J. Cognitive Neurosci. 3 1 (winter 1991) 71-86
15. Belhumeur P.N., Hespanha J.P., and Kriegman D.J. Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. 19 7 (1997) 711-720
16. Pontil M., and Verri A. Support vector machines for 3d object recognition. IEEE Trans. Pattern Anal. 20 6 (1998) 637-646
17. Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D., and Lander E.S. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286 5439 (1999) 531-537
18. Tusher V.G., Tibshirani R., and Chu G. Significance analysis of microarrays applied to the ionizing radiation response. Proc. Natl. Acad. Sci. USA 98 9 (2001) 5116-5121
19. Rizzuto D.S., Mamelak A.N., Sutherling W.W., Fineman I., and Andersen R.A. Spatial selectivity in human ventrolateral prefrontal cortex. Nat. Neurosci. 8 (2005) 415-417
20. Nenadic Z., Rizzuto D.S., Andersen R.A., and Burdick J.W. Advances in cognitive neural prosthesis: recognition of neural data with an information-theoretic objective. In: Dornhege G., Millan J.R., Hinterberger T., McFarland D.J., and Muller K.-R. (Eds). Toward Brain Computer Interfacing (2007), MIT Press, Cambridge, MA 175-190
21. Huber P.J. Projection pursuit. Ann. Stat. 13 2 (1985) 435-475
22. Schäfer J., and Strimmer K. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat. Appl. Genet. Mol. Biol. 4 1 (2005) (Article 32)
23. Daniels M.J., and Kass R.E. Shrinkage estimators for covariance matrices. Biometrics 57 (2001) 1173-1184
24. Hastie T., Tibshirani R., and Friedman J. The Elements of Statistical Learning (2001), Springer Series in Statistics
25. E.P. Xing, M.I. Jordan, R.M. Karp, Feature selection for high-dimensional genomic microarray data, in: ICML '01: Proceedings of the Eighteenth International Conference on Machine Learning, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc., 2001, pp. 601-608.
26. C.E. Thomaz, D.F. Gillies, R.Q. Feitosa, Small sample problem in Bayes plug-in classifier for image recognition, in: International Conference on Image and Vision Computing New Zealand, 2001, pp. 295-300.
27. Hoffbeck J.P., and Landgrebe D.A. Covariance matrix estimation and classification with limited training data. IEEE Trans. Pattern Anal. 18 7 (1996) 763-767
28. Jolliffe I.T. Principal Component Analysis (1986), Springer, New York
29. Chen L.-F., Liao H.-Y.M., Ko M.-T., Lin J.-C., and Yu G.-J. A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognition 33 10 (2000) 1713-1726
30. Yu H., and Yang H. A direct LDA algorithm for high-dimensional data-with application to face recognition. Pattern Recognition Lett. 34 10 (2001) 2067-2070
31. Wang X., and Tang X. Dual-space linear discriminant analysis for face recognition. 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04) 02 (2004) 564-569
32. Zhu M., and Martinez A.M. Subclass discriminant analysis. IEEE Trans. Pattern Anal. 28 8 (2006) 1274-1286
33. Kirby M., and Sirovich L. Application of the Karhunen-Loeve procedure for the characterization of human faces. IEEE Trans. Pattern Anal. 12 (1990) 103-108
34. R. Huang, Q. Liu, H. Lu, S. Ma, Solving the small sample size problem of lda, in: ICPR '02: Proceedings of the 16th International Conference on Pattern Recognition (ICPR'02), vol. 3, 2002, pp. 29-32.
35. Zhu M., and Martinez A.M. Selecting principal components in a two-stage lda algorithm. CVPR '06: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2006), IEEE Computer Society, Washington, DC, USA 132-137
36. Zhang S., and Sim T. Discriminant subspace analysis: a Fukunaga-Koontz approach. IEEE Trans. Pattern Anal. 29 10 (2007) 1732-1745
37. Gao H., and Davis J.W. Why direct lda is not equivalent to lda. Pattern Recognition 39 5 (2006) 1002-1006
38. Fukunaga K., and Koontz W.L.G. Application of the Karhunen-Loeve expansion to feature selection and ordering. IEEE Trans. Comput. C-19 4 (1970) 311-318
39. Schölkopf B., Smola A., and Müller K.-R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10 5 (1998) 1299-1319
40. Baudat G., and Anouar F. Generalized discriminant analysis using a kernel approach. Neural Comput. 12 10 (2000) 2385-2404
41. S. Mika, G. Ratsch, J. Weston, B. Scholkopf, K.-R. Müller, Fisher discriminant analysis with kernels, in: Proceedings of IEEE Neural Networks for Signal Processing Workshop, 1999, pp. 41-48.
42. Mika S., Weston J., and Smola A. Invariant feature extraction and classification in kernel spaces. In: Solla S.A., Leen T.K., and Muller K.-R. (Eds). Advances in Neural Information Processing Systems 12 (2000), MIT Press, Cambridge, MA 526-532
43. Mika S., Rätsch G., Weston J., Schölkopf B., Smola A., and Müller K.-R. Constructing descriptive and discriminative nonlinear features: Rayleigh coefficients in kernel feature spaces. IEEE Trans. Pattern Anal. 25 5 (2003) 623-633
44. Yang M. Kernel eigenfaces vs. kernel fisherfaces: face recognition using kernel methods. FGR '02: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (2002), IEEE Computer Society, Washington, DC, USA 215
45. Yang J., Frangi A.F., Yang J.-Y., and Jin Z. KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition. IEEE Trans. Pattern Anal. 27 2 (2005) 230-244
46. Roweis S.T., and Saul L.K. Nonlinear dimensionality reduction by locally linear embedding. Science 290 (2000) 2323-2326
47. Tenenbaum J.B., de Silva V., and Langford J.C. A global geometric framework for nonlinear dimensionality reduction. Science 290 (2000) 2319-2323
48. He X., Yan S., Hu Y., Niyogi P., and Zhang H.-J. Face recognition using laplacianfaces. IEEE Trans. Pattern Anal. 27 3 (2005) 328-340
49. Y. Fu, T.S. Huang, Locally linear embedded eigenspace analysis, U.I.U.C. Technical Report IFP-TR, ECE, 2005.
50. H.-T. Chen, H.-W. Chang, T.-L. Liu, Local discriminant embedding and its variants, in: Proceedings of Computer Vision and Pattern Recognition, CVPR, vol. 2, June 2005, pp. 846-853.
51. Yan S., Xu D., Zhang B., Zhang H.-J., Yang Q., and Lin S. Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans. Pattern Anal. 29 1 (2007) 40-51
52. Fu Y., Yan S., and Huang T.S. Classification and feature extraction by simplexization. IEEE Trans. Inf. Forensics Secur. 3 1 (2008) 91-100
53. Fu Y., and Huang T.S. Image classification using correlation tensor analysis. IEEE Trans. Image Process. 17 2 (2008) 226-234
54. Xiaofei H., and Niyogi P. Locality preserving projections. Advances in Neural Information Processing Systems (NIPS) vol. 18 (2003), MIT Press, Cambridge, MA
55. Vasilescu M.A.O., and Terzopoulos D. Multilinear subspace analysis of image ensembles. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2 (2003) p. II-93-99
56. Ye J., Janardan R., and Li Q. Two-Dimensional Linear Discriminant Analysis. In: Saul L.K., Yair W., and Léon B. (Eds). Advances in Neural Information Processing Systems 17 (2005), MIT Press, Cambridge, MA 1569-1576
57. He X., Cai D., and Niyogi P. Tensor subspace analysis. Advances in Neural Information Processing Systems (NIPS) vol. 18 (2006), MIT Press, Cambridge, MA 499-506
58. K. Das, J. Meyer, Z. Nenadic, Analysis of large-scale brain data for brain-computer interfaces, in: Proceedings of the 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2006, pp. 5731-5734.
59. K. Das, S. Osechinskiy, Z. Nenadic, A classwise pca-based recognition of neural data for brain-computer interfaces, in: Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2007, pp. 6519-6522.
60. Meyer F.G., and Shen X. Classification of fmri time series in a low-dimensional subspace with a spatial prior. IEEE Trans. Med. Imaging 27 1 (2008) 87-98
61. Wolpaw J.R., and McFarland D.J. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. USA 101 51 (2004) 17849-17854
62. Pfurtscheller G., Neuper C., Flotzinger D., and Pregenzer M. EEG-based discrimination between imagination of right and left hand movement. Electroen. Clin. Neuro. 103 6 (1997) 642-651
63. S. Shan, W. Gao, X. Chen, J. Ma, Novel face recognition based on individual eigen-subspaces, in: Signal Processing Proceedings, 2000, 5th International Conference on WCCC-ICSP 2000, vol. 3, 21-25 August 2000, pp. 1522-1525.
64. Shan S.G., Gao W., and Zhao D. Face recognition based on face-specific subspace. Int. J. Imaging Syst. Technol. 13 1 (2003) 23-32
65. Liu Z.-Y., Chiu K.-C., and Xu L. Improved system for object detection and star/galaxy classification via local subspace analysis. Neural Networks 16 3-4 (2003) 437-451
66. Min W., Lu K., and He X. Locality pursuit embedding. Pattern Recognition 37 4 (2004) 781-788
67. Mallat S. A Wavelet Tour of Signal Processing. second ed. (1999), Academic Press, Inc., New York
68. Donoho D.L. De-noising by soft thresholding. IEEE Trans. Inf. Theory 41 3 (1995) 613-627
69. Parzen E. On the estimation of a probability density function and mode. Ann. Math. Stat. 33 (1962) 1065-1076
70. Tubbs J.D., Coberly W.A., and Young D.M. Linear dimension reduction and Bayes classification with unknown population parameters. Pattern Recognition 15 3 (1982) 167-172
71. Brunzell H., and Eriksson J. Feature reduction for classification of multidimensional data. Pattern Recognition 33 10 (2000) 1741-1748
72. S. Hettich, C.L. Blake, C.J. Merz, UCI repository of machine learning databases 〈http://www.ics.uci.edu/∼mlearn/MLRepository.html〉, Department of Information and Computer Sciences, University of California, Irvine, 1998.
73. Kittler J., Hatef M., Duin R.P.W., and Matas J. On combining classifiers. IEEE Trans. Pattern Anal. 20 3 (1998) 226-239
74. Michalski R.S. Learning by being told and learning from examples: an experimental comparison of the two methods of knowledge acquisition in the context of developing an expert system for soybean disease diagnosis. Int. J. Pol. Anal. Inf. Syst. 4 2 (1980) 125-161
75. Alon U., Barkai N., Notterman D.A., Gish K., Ybarra S., Mack D., and Levine A.J. Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc. Natl. Acad. Sci. USA 96 12 (1999) 6745-6750
76. Petricoin E.F., Ardekani A.M., Hitt B.A., Levine P.J., Fusaro V.A., Steinberg S.M., Mills G.B., Simone C., Fishman D.A., Kohn E.C., and Liotta L.A. Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359 9306 (2002) 572-577
77. Torkkola K. Feature extraction by non-parametric mutual information maximization. J. Mach. Learn. Res. 3 (2003) 1415-1438
78. R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in: International Joint Conference on Artificial Intelligence, 1995, pp. 1137-1145.