Application of global optimization methods to model and feature selection

Pattern Recognition

Volumn 45, Issue 10, 2012, Pages 3676-3686

  Boubezoul, Abderrahmane ^a   Paris, Sébastien ^b  

^a INRETS   (France)

^b AIX MARSEILLE UNIVERSITY   (France)

Author keywords

Cross Entropy Method; Feature selection; Hyper parameters optimization; Particle swarm optimization; Support vector machines

Indexed keywords

CROSS-ENTROPY METHOD; DATA MINING APPLICATIONS; EXHAUSTIVE SEARCH; FEATURE SUBSET; FINITE SUBSETS; GENERALIZATION PERFORMANCE; GLOBAL OPTIMIZATION METHOD; INDUCTION ALGORITHMS; LARGE DATASETS; OPTIMIZATION APPROACH; OPTIMIZATION PROBLEMS; OPTIMIZATION PROCEDURES; PARAMETER VALUES; PREDICTION MODEL;

FEATURE EXTRACTION; GLOBAL OPTIMIZATION; MATHEMATICAL MODELS; PARTICLE SWARM OPTIMIZATION (PSO); SUPPORT VECTOR MACHINES;

OPTIMIZATION;

EID: 84861820472     PISSN: 00313203     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patcog.2012.04.015     Document Type: Article

References (49)

1. J. Bekker, and C. Aldrich The cross-entropy method in multi-objective optimisation: an assessment European Journal of Operational Research 211 May (1) 2011 112 121
2. A. Blum, and P. Langley Selection of relevant features and examples in machine learning Artificial Intelligence 97 1997 245 271
3. P.D. Boer, D. Kroese, S. Mannor, and R.Y. Rubinstein A tutorial on the cross-entropy method Annals of Operations Research 134 2005 19 67
4. R. Caruana, D. Freitag, Greedy attribute selection, in: International Conference on Machine Learning, 1994, pp. 28-36.
5. K. Chepuri, and T.H. de Mello Solving the vehicle routing problem with stochastic demands using the cross entropy method Annals of Operations Research 134 2005 153 181
6. L.-Y. Chuang, H.-W. Chang, C.-J. Tu, and C.-H. Yang Improved binary PSO for feature selection using gene expression data Computational Biology and Chemistry 32 1 2008 29 38
7. M. Clerc, and J. Kennedy The particle swarm-explosion, stability, and convergence in a multidimensional complex space IEEE Transaction on Evolutionary Computation 6 1 2002 58 73
8. E.S. Correa, M.T. Steiner, A.A. Freitas, and C. Carnieri Using a genetic algorithm for solving a capacity p-median problem Numerical Algorithms 35 2004 373 388
9. C. Cortes, V. Vapnik, Support-vector networks, Machine Learning 20 (3) (1995) 273-297, http://dx.doi.org/10.1023/A:1022627411411.
10. T. Cover, P. Hart, Nearest neighbor pattern classification, IEEE Transactions on Information Theory, 13 (1) (1967) 21-27, http://dx.doi.org/10. 1109/TIT.1967.1053964
11. K. Crammer, Y. Singer, On the learnability and design of output codes for multiclass problems, in: The Thirteenth Annual Conference on Computational Learning Theory, 2000, pp. 35-46.
12. G.V. Dijck, M.M.V. Hulle, and M. Wevers Genetic algorithm for feature subset selection with exploitation of feature correlations from continuous wavelet transform: a real-case application A. Okatan, International Conference on Computational Intelligence 2004 International Computational Intelligence Society 34 38
13. R. Eberhart, J. Kennedy, A new optimizer using particle swarm theory, in: Proceedings of the Sixth International Symposium on Micromachine and Human Science, 1995, pp. 39-43.
14. R.C. Eberhart, Y. Shi, and J. Kennedy Swarm Intelligence, The Morgan Kaufmann Series in Artificial Intelligence 2001 Morgan Kaufmann
15. A. Freitas Data Mining and Knowledge Discovery with Evolutionary Algorithms October 2002 Springer-Verlag
16. H. Frohlich, O. Chapelle, Feature selection for support vector machines by means of genetic algorithms, in: Proceedings of the 15th IEEE International Conference on Tools with Artificial Intelligence, vol. 12, 2003, pp. 142-148.
17. M.C. Fu, F.W. Glover, J. April, Simulation optimization: a review, new developments, and applications, in: The 37th Winter Simulation Conference (WSC2005), 2005, pp. 83-95.
18. Y. Guermeur, A. Eliseeff, H. PaugamMoisy, A new multi-class SVM based on a uniform convergence result, in: The IEEE-INNS-ENNS International Joint Conference on Neural Networks IJCNN 2000, vol. IV, 2000, pp. 183-188.
19. I. Guyon, and A. Elisseeff An introduction to variable and feature selection Journal of Machine Learning Research 3 2003 1157 1182
20. B. Hammer, and K. Gersmann A note on the universal approximation capability of support vector machines Neural Processing Letters 17 1 2003 43 53
21. S.S. Keerthi, and C.-J. Lin Asymptotic behaviors of support vector machines with Gaussian kernel Neural Computation 15 2003 1667 1689
22. J. Kennedy, R.C. Eberhart, A discrete binary version of the particle swarm algorithm, in: Proceedings of the 1997 Conference on Systems, Man, and Cybernetics, 1997, pp. 4104-4109.
23. K. Kira, L.A. Rendell, The feature selection problem: traditional methods and a new algorithm, in: Proceedings of AAAI-92, 1992, pp. 129-134.
24. R. Kohavi, and G.H. John Wrappers for feature subset selection Artificial Intelligence 97 1997 273 324
25. D. Koller, M. Sahami, Toward optimal feature selection, in: ICML, 1996, pp. 284-292.
26. I. Kononenko, Estimating attributes: analysis and extensions of relief, in: European Conference on Machine Learning, 1994, pp. 171-182.
27. D. Kroese, S. Porotsky, and R. Rubinstein The cross-entropy method for continuous multi-extremal optimization Methodology and Computing in Applied Probability 8 2006 383 407
28. H.-T. Lin, C.-J. Lin, A Study on Sigmoid Kernels for SVM and the Training of Non-PSD Kernels by SMO-type Methods, Technical Report, Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, 2003.
29. Z. Liu, A. Doucet, S.S. Singh, The cross-entropy method for blind multiuser detection, in: IEEE International Symposium on Information Theory, 2004.
30. R. Meiri, and J. Zahavi Using simulated annealing to optimize the feature selection problem in marketing applications European Journal of Operational Research 171 June (3) 2006 842 858
31. D.J. Newman, S. Hettich, C.L. Blake, C.J. Merz, UCI Repository of Machine Learning Databases. 〈 http://www.ics.uci.edu/∼mlearn/MLRepository.html 〉, 1998.
32. J. Novovicová, P. Pudil, and J. Kittler Divergence based feature selection for multimodal class densities IEEE Transactions on Pattern Analysis and Machine Intelligence 18 February 1996 218 223
33. K.E. Parsopoulos, and M.N. Vrahatis Recent approaches to global optimization problems through particle swarm optimization Natural Computing 1 June (2) 2002 235 306
34. T. Peters, D.W. Bulger, T.H. Loi, J.Y.H. Yang, and D. Ma Two-step cross-entropy feature selection for microarrays -power through complementarity IEEE/ACM Transactions on Computational Biology and Bioinformatics 8 July (4) 2011 1148 1151
35. P. Pudil, J. Novovicova, and J. Kittler Floating search methods in feature-selection Pattern Recognition Letters 15 November (11) 1994 1119 1125
36. R. Rubinstein Optimization of computer simulation models with rare events European Journal of Operational Research 99 1997 89 112
37. R. Rubinstein The cross-entropy method for combinatorial and continuous optimization Methodology and Computing in Applied Probability 1 1999 127 190
38. R.Y. Rubinstein, and D.P. Kroese The Cross-Entropy Method: A Unified Approach to Combinatorial Method, Monte-Carlo Simulation, Randomized Optimization and Machine Learning 2004 Springer Verlag
39. Y. Shi, R.C. Eberhart, A modified particle swarm optimizer, in: Proceedings of the IEEE International Conference on Evolutionary Computation, 1998, IEEE Press, Piscataway, NJ, pp. 69-73.
40. Y. Shi, R.C. Eberhart, shi99-cec.pdf, Empirical study of particle swarm optimization, in: Proceedings of 1999 Congress on Evolutionary Computation, July 6-9 1999, pp. 1945-1950.
41. M. Stone Cross validation choice and assessment of statistical predictions Journal of the Royal Statistical Society B 36 1974 111 147
42. A. Ünveren, A. Acan, Multi-objective optimization with cross entropy method: stochastic learning with clustered Pareto fronts, In: IEEE Congress on Evolutionary Computation, 2007, pp. 3065-3071.
43. V.N. Vapnik Statistical Learning Theory 1998 Wiley
44. X. Wang, J. Yang, X. Teng, W. Xia, and R. Jensen Feature selection based on rough sets and particle swarm optimization Pattern Recognition Letters 28 4 2007 459 471
45. C.W. Hsu, C.C. Chang, C.J. Lin, A practical Guide to Support Vector Classification, Technical Report, Department of Computer Science, National Taiwan University, 2010.
46. J. Weston, C. Watkins, Multi-class Support Vector Machines, Technical Report, Department of Computer Science, Royal Holloway, University of London, Egham, UK, 2000.
47. O. Wittner, B.E. Helvik, Cross entropy guided ant-like agents finding dependable primary/backup path patterns in networks, in: Congress on Evolutionary Computation (CEC2002), 2002.
48. J. Yang, and V. Honavar Feature subset selection using a genetic algorithm IEEE Intelligent Systems 13 1998 44 49
49. Z. Zhen, X. Zeng, H. Wang, L. Han, A global evaluation criterion for feature selection in text categorization using Kullback-Leibler divergence, in: 2011 International Conference of Soft Computing and Pattern Recognition (SoCPaR), 2011, pp. 440-445.