Resampling methods for quality assessment of classifier performance and optimal number of features

Signal Processing

Volumn 93, Issue 11, 2013, Pages 2956-2968

  Fandos, Raquel ^a   Debes, Christian ^b   Zoubir, Abdelhak M ^a  

^a DARMSTADT UNIVERSITY OF TECHNOLOGY   (Germany)

^b AGT Group (RandD) GmbH   (Germany)

Author keywords

Classifier design and evaluation; Feature evaluation and selection; Optimal dimensionality; Pattern recognition; Resampling Bootstrap

Indexed keywords

CLASSIFICATION SYSTEM; CLASSIFIER DESIGN AND EVALUATION; CLASSIFIER PERFORMANCE; FEATURE EVALUATION AND SELECTION; FEATURE SELECTION ALGORITHM; MISCLASSIFICATION RATES; OPTIMAL DIMENSIONALITY; RESAMPLING;

ALGORITHMS; PATTERN RECOGNITION; PROBABILITY DISTRIBUTIONS; STATISTICAL METHODS;

OPTIMIZATION;

EID: 84879317733     PISSN: 01651684     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.sigpro.2013.05.004     Document Type: Article

References (57)

1. A.K. Jain, R.P.W. Duin, and M. Jianchang Statistical pattern recognition a review IEEE Transactions on Pattern Analysis and Machine Intelligence 22 1 2000 4 37
2. C.M. Bishop Pattern Recognition and Machine Learning (Information Science and Statistics) 2006 Springer-Verlag New York, Inc. Secaucus, NJ, USA
3. B.D. Ripley Pattern Recognition and Neural Networks 1996 Cambridge University Press Cambridge
4. T.W. Anderson An Introduction to Multivariate Statistical Analysis 3rd ed. 2003 John Wiley and Sons, Inc. Hoboken, NJ
5. L. Breiman, J.H. Friedman, and R.A. Olshen Classification and Regression Trees 1984 Wadsworth
6. C. Cortes, and V. Vapnik Support-vector networks Machine Learning 20 3 1995 273 297
7. Q.-L. Tran, K.-A. Toh, D. Srinivasan, K.-L. Wong, and S.Q.-C. Low An empirical comparison of nine pattern classifiers IEEE Transactions on Systems, Man, and Cybernetics, Part B Cybernetics 35 5 2005 1079 1091
8. M.P. Sampat, A.C. Patel, Y. Wang, S. Gupta, C.-W. Kan, A.C. Bovik, and M.K. Markey Indexes for three-class classification performance assessment - an empirical comparison IEEE Transactions on Information Technology in Biomedicine 13 3 2009 300 312
9. W.A. Yousef, R.F. Wagner, M.H. Loew, Comparison of non-parametric methods for assessing classifier performance in terms of ROC parameters, in: Proceedings of the Applied Imagery Pattern Recognition Workshop, 2004, pp. 190-195.
10. W.E. Weideman, M.T. Manry, H.-C. Yau, and W. Gong Comparisons of a neural network and a nearest-neighbor classifier via the numeric handprint recognition problem IEEE Transactions on Neural Networks 6 6 1995 1524 1530
11. J.D. Paola, and R.A. Schowengerdt A detailed comparison of backpropagation neural network and maximum-likelihood classifiers for urban land use classification IEEE Transactions on Geoscience and Remote Sensing 33 4 1995 981 996
12. R.E. Bellman, Adaptive Control Processes, Princeton University Press, 1961
13. B. Kim, D.A. Landgrebe, Prediction of optimal number of features, in: Proceedings of the Annual International Geoscience and Remote Sensing Symposium, 1990, pp. 2393-2396.
14. A.M. Zoubir, and D.R. Iskander Bootstrap Techniques for Signal Processing 2004 Cambridge University Press
15. A.K. Jain, R.C. Dubes, and C. Chen Bootstrap techniques for error estimation IEEE Transactions on Pattern Analysis and Machine Intelligence 5 1987 628 633
16. S. Borra, and A. Di Ciaccio Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods Computational Statistics & Data Analysis 54 2010 2976 2989
17. H.-C. Chung, and C.-P. Han Conditional confidence intervals for classification error rate Computational Statistics & Data Analysis 53 12 2009 4358 4369
18. J. Dias, and J. Vermunt A bootstrap-based aggregate classifier for model-based clustering Computational Statistics 23 4 2008 643 659
19. A. Isaksson, M. Wallman, H. Göransson, and M.G. Gustafsson Cross-validation and bootstrapping are unreliable in small sample classification Pattern Recognition Letters 29 2008 1960 1965
20. N. Diaz-Diaz, J.S. Aguilar-Ruiz, J.A. Nepomuceno, J. Garcia, Feature selection based on bootstrapping, in: Congress on Computational Intelligence Methods and Applications, ICSC, 2005.
21. M. Verleysen, F. Rossi, D. François, Advances in feature selection with mutual information, Similarity-based Clustering, Springer-Verlag, Berlin, Heidelberg, 2009, pp. 52-69.
22. J.-L. Yuan, Bootstrapping nonparametric feature selection algorithms for mining small data sets, in: International Joint Conference on Neural Networks, vol. 4, 1999, pp. 2526-2529.
23. A.M. Zoubir, and R. Iskander Bootstrap methods and applications IEEE Signal Processing Magazine 24 4 2007 10 19
24. D.W.K. Andrews, and M. Buchinsky A three-step method for choosing the number of bootstrap repetitions Econometrica 68 1 2000 23 51
25. J. Shao, and D. Tu The Jackknife and Bootstrap 1995 Springer-Verlag New York, Inc.
26. D.N. Politis, J.P. Romano, and M. Wolf Subsampling (Springer Series in Statistics) 1999 Springer-Verlag New York, Inc.
27. M. Abramowitz, and I.A. Stegun Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables 1972 Dover Publications
28. A.W. Whitney A direct method of nonparametric measurement selection IEEE Transactions on Computers 20 1971 1100 1103
29. G. Hughes On the mean accuracy of statistical pattern recognizers IEEE Transactions on Information Theory 14 1 1968 55 63
30. C.R. Rao On some problems arising out of discrimination with multiple characters Sankhya The Indian Journal of Statistics (1933-1960) 9 4 1949 343 366
31. A.K. Jain, and B. Chandrasekaran Dimensionality and sample size considerations in pattern recognition practice P. Krishnaiah, L. Kanal, Classification Pattern Recognition and Reduction of Dimensionality, Handbook of Statistics vol. 2 1982 Elsevier 835 855
32. T.L. Boullion, P.L. Odell, and B.S. Duran Estimating the probability of misclassification and variate selection Pattern Recognition 7 3 1975 139 145
33. J. van Ness On the effects of dimension in discriminant analysis Technometrics 18 1979 175 187
34. A.K. Jain, and R. Duber On the optimal number of features in the classification of multivariate Gaussian data Pattern Recognition 10 1978 365 374
35. S. Raudys, and V. Pikelis On dimensionality, sample size, classification error, and complexity of classification algorithm in pattern recognition IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-2 3 1980 242 252
36. A. Bhattacharyya On a measure of divergence between two statistical populations defined by their probability distributions Bulletin of the Calcutta Mathematical Society 35 1943 99 109
37. T.K. Ho The random subspace method for constructing decision forests IEEE Transactions on Pattern Analysis and Machine Intelligence 20 8 1998 832 844
38. T.M. Cover, and J.M. Van Campenhout On the possible orderings in the measurement selection problem IEEE Transactions on Systems, Man and Cybernetics 7 9 1977 657 661
39. P. Pudil, J. Novovičová, and J. Kittler Floating search methods in feature selection Pattern Recognition Letters 15 1994 1119 1125
40. H. Zou, T. Hastie, and R. Tibshirani Sparse principal component analysis Journal of Computational and Graphical Statistics 15 2006 262 286
41. J. Li, and D. Tao On preserving original variables in Bayesian pca with application to image analysis IEEE Transactions on Image Processing 21 12 2012 4830 4843
42. H. Ishwaran, and J.S. Rao Spike and slab variable selection frequentist and Bayesian strategies The Annals of Statistics 33 2 2005 730 773
43. J. Hua, W.D. Tembe, and E.R. Dougherty Performance of feature-selection methods in the classification of high-dimension data Pattern Recognition 42 3 2009 409 424
44. I.A. Gheyas, and L.S. Smith Feature subset selection in large dimensionality domains Pattern Recognition 43 1 2010 5 13
45. S.-W. Lin, K.-C. Ying, S.-C. Chen, and Z.-J. Lee Particle swarm optimization for parameter determination and feature selection of support vector machines Expert Systems with Applications 35 4 2008 1817 1824
46. H. Zeng, and Y.-M. Cheung A new feature selection method for Gaussian mixture clustering Pattern Recognition 42 2 2009 243 250 Learning Semantics from Multimedia Content
47. Z. Xu, I. King, M.-T. Lyu, and R. Jin Discriminative semi-supervised feature selection via manifold regularization IEEE Transactions on Neural Networks 21 7 2010 1033 1047
48. H. Liu, J. Sun, L. Liu, and H. Zhang Feature selection with dynamic mutual information Pattern Recognition 42 7 2009 1330 1339
49. Y. Sun, S. Todorovic, and S. Goodison Local-learning-based feature selection for high-dimensional data analysis IEEE Transactions on Pattern Analysis and Machine Intelligence 32 9 2010 1610 1626
50. J.-B. Yang, and C.-J. Ong Feature selection using probabilistic prediction of support vector regression IEEE Transactions on Neural Networks 22 6 2011 954 962
51. A.K. Jain, and D. Zongker Feature selection evaluation, application, and small sample performance IEEE Transactions on Pattern Analysis and Machine Intelligence 19 2 1997 153 158
52. C. Sima, S. Attoor, U. Brag-Neto, E.R.D.J. Lowey, and E. Suh Impact of error estimation on feature selection Pattern Recognition 38 12 2005 2472 2482
53. S. Kullback, and R.A. Leibler On information and sufficiency Annals of Mathematical Statistics 22 1951 79 86
54. A. Frank, A. Asuncion, UCI Machine Learning Repository, 2010.
55. R. Fandos, A.M. Zoubir, K. Siantidis, Unified design of a feature based ADAC system for mine hunting using synthetic aperture sonar, IEEE Transactions on Geoscience and Remote Sensing, accepted for publication.
56. C.C. Chang, C.J. Lin, LIBSVM: A Library for Support Vector Machines, 2001.
57. E. Parzen On estimation of a probability density function and mode Annals of Mathematical Statistics 33 1962 1065 1076