Wrappers for feature subset selection

Artificial Intelligence

Volumn 97, Issue 1-2, 1997, Pages 273-324

  Kohavi, Ron ^a   John, George H ^b  

^a HEWLETT PACKARD ENTERPRISE   (United States)

^b Epiphany Marketing Software   (United States)

Author keywords

Classification; Feature selection; Filter; Wrapper

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DATA STRUCTURES; DECISION THEORY; FORMAL LOGIC; LEARNING ALGORITHMS; TREES (MATHEMATICS);

FEATURE SELECTION; WRAPPERS;

LEARNING SYSTEMS;

EID: 0031381525     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/s0004-3702(97)00043-x     Document Type: Article

References (123)

1. D.W. Aha, Tolerating noisy, irrelevant and novel attributes in instance-based learning algorithms, Internat. J. Man-Machine Studies 36 (1992) 267-287.
2. D.W. Aha and R.L. Bankert, Feature selection for case-based classification of cloud types: an empirical comparison, in: Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, Seattle, WA (1994) 106-112.
3. D.W. Aha and R.L. Bankert, A comparative evaluation of sequential feature selection algorithms, in: D. Fisher and H. Lenz, eds., Proceedings 5th International Workshop on Artificial Intelligence and Statistics, Ft. Lauderdale, FL (1995) 1-7.
4. D.W. Aha, D. Kibler and M.K. Albert, Instance-based learning algorithms, Machine Learning 6 (1991) 37-66.
5. H. Almuallim and T.G. Dietterich, Learning with many irrelevant features, in: Proceedings AAAI-91, Anaheim, CA (MIT Press, Cambridge, MA, 1991) 547-552.
6. H. Almuallim and T.G. Dietterich, Learning Boolean concepts in the presence of many irrelevant features, Artificial Intelligence 69 (1994) 279-306.
7. J.R. Anderson and M. Matessa, Explorations of an incremental, Bayesian algorithm for categorization, Machine Learning 9 (1992) 275-308.
8. C.G. Atkeson, Using locally weighted regression for robot learning, in Proceedings IEEE International Conference on Robotics and Automation, Sacramento, CA (1991) 958-963.
9. J. Bala, K.A.D. Jong, J. Haung, H. Vafaie and H. Wechsler, Hybrid learning using genetic algorithms and decision trees for pattern classification, in: C.S. Mellish, ed., Proceedings IJCAI-95, Montreal, Que. (Morgan Kaufmann, Los Altos, CA, 1995) 719-724.
10. M. Ben-Bassat, Use of distance measures, information measures and error bounds in feature evaluation, in: P.R. Krishnaiah and L.N. Kanal, eds., Handbook of Statistics, Vol. 2 (North-Holland, Amsterdam, 1982) 773-791.
11. H. Berliner, The B* tree search algorithm: a best-first proof procedure, Artificial Intelligence 12 (1979) 23-40; reprinted in: B. Webber and N.J. Nilsson, eds., Readings in Artificial Intelligence (Morgan Kaufmann, Los Altos, CA, 1981) 79-87.
12. H. Berliner, The B* tree search algorithm: a best-first proof procedure, Artificial Intelligence 12 (1979) 23-40; reprinted in: B. Webber and N.J. Nilsson, eds., Readings in Artificial Intelligence (Morgan Kaufmann, Los Altos, CA, 1981) 79-87.
13. A.L. Blum and R.L. Rivest, Training a 3-node neural network is NP-complete, Neural Networks 5 (1992) 117-127.
14. M. Boddy and T. Dean, Solving time-dependent planning problems, in: N.S. Sridharan, ed., Proceedings IJCAI-89, Detroit, MI (Morgan Kaufmann, Los Altos, CA, 1989) 979-984.
15. P. Brazdil, J. Gama and B. Henery, Characterizing the applicability of classification algorithms using meta-level learning, in: F. Bergadano and L.D. Raedt, eds., Proceedings European Conference on Machine Learning (1994).
16. L. Breiman, Bagging predictors, Machine Learning 24 (1996) 123-140.
17. L. Breiman, J.H. Friedman, R.A. Olshen and C.J. Stone, Classification and Regression Trees (Wadsworth, Belmont, CA, 1984).
18. W. Buntine, Learning classification trees, Statist. and Comput. 2 (1992) 63-73.
19. C. Cardie, Using decision trees to improve case-based learning, in: Proceedings 10th International Conference on Machine Learning, Amherst, MA (Morgan Kaufmann, Los Altos, 1993) 25-32.
20. R. Caruana and D. Freitag, Greedy attribute selection, in: W.W. Cohen and H. Hirsh, eds., Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, Los Altos, CA, 1994) 28-36.
21. B. Cestnik, Estimating probabilities: a crucial task in machine learning, in: L.C. Aiello, ed., Proceedings ECAI-90, Stockholm, Sweden (1990) 147-149.
22. T.M. Cover and J.M.V. Campenhout, On the possible orderings in the measurement selection problem, IEEE Trans. Systems Man Cybernet. 7 (1977) 657-661.
23. B.V. Dasarathy, Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques (IEEE Computer Society Press, Los Alamitos, CA, 1990).
24. R.L. De Mántaras, A distance-based attribute selection measure for decision tree induction, Machine Learning 6 (1991) 81-92.
25. P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach (Prentice-Hall, Englewood, Cliffs, NJ, 1982).
26. J. Doak, An evaluation of feature selection methods and their application to computer security, Tech. Rept. CSE-92-18, University of California at Davis, CA (1992).
27. P. Domingos and M. Pazzani, Beyond independence: conditions for the optimality of the simple Bayesian classifier, in: L. Saitta, ed., Proceedings 13th International Conference on Machine Learning, Bari, Italy (Morgan Kaufmann, Los Altos, CA, 1996) 105-112.
28. J. Dougherty, R. Kohavi and M. Sahami, Supervised and unsupervised discretization of continuous features, in: A. Prieditis and S. Russell, eds., Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, Los Altos, CA, 1995) 194-202.
29. N.R. Draper and H. Smith, Applied Regression Analysis (Wiley, New York, 2nd ed., 1981).
30. R. Duda and P. Hart, Pattern Classification and Scene Analysis (Wiley, New York, 1973).
31. U.M. Fayyad, On the induction of decision trees for multiple concept learning, Ph.D. Thesis, EECS Department, Michigan University (1991).
32. U.M. Fayyad and K.B. Irani, The attribute selection problem in decision tree generation, in: Proceedings AAAI-92, San Jose, CA (MIT Press, Cambridge, MA, 1992) 104-110.
33. P.W.L. Fong, A quantitative study of hypothesis selection, in: A. Prieditis and S. Russell, eds., Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, Los Altos, CA, 1995) 226-234.
34. Y. Freund, Boosting a weak learning algorithm by majority, in: Proceedings 3rd Annual Workshop on Computational Learning Theory, San Francisco, CA (1990) 202-216; also: Inform. and Comput., to appear.
35. Y. Freund and R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, in: Proceedings 2nd European Conference on Computational Learning Theory (Springer, Berlin, 1995) 23-37.
36. G.M. Furnival and R.W. Wilson, Regression by leaps and bounds, Technometrics 16 (1974) 499-511.
37. S. Geman, E. Bienenstock and R. Doursat, Neural networks and the bias/variance dilemma, Neural Comput. 4 (1992) 1-48.
38. J.H. Gennari, P. Langley and D. Fisher, Models of incremental concept formation, Artificial Intelligence 40 (1989) 11-61.
39. M.L. Ginsberg, Essentials of Artificial Intelligence (Morgan Kaufmann, Los ALtos, CA, 1993).
40. D.E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning (Addison-Wesley, Reading, MA, 1989).
41. I.J. Good, The Estimation of Probabilities: An Essay on Modern Bayesian Methods (MIT Press, Cambridge, MA, 1965).
42. R. Greiner, Probabilistic hill climbing: theory and applications, in: J. Glasgow and R. Hadley, eds., Proceedings 9th Canadian Conference on Artificial Intelligence, Vancouver, BC (Morgan Kaufmann, Los Altos, CA, 1992) 60-67.
43. T.R. Hancock, On the difficulty of finding small consistent decision trees, Unpublished manuscript, Harvard University, Cambridge, MA (1989).
44. W. Hoeffding, Probability inequalities for sums of bounded random variables, J. Amer. Statist. Assoc. 58 (1963) 13-30.
45. J.H. Holland, Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control and Artificial Intelligence (MIT Press, Cambridge, MA, 1992).
46. L. Hyafil and R.L. Rivest, Constructing optimal binary decision trees is NP-complete, Inform. Process. Lett. 5 (1976) 15-17.
47. G.H. John, Enhancements to the data mining process, Ph.D. Thesis, Computer Science Department, Stanford University, CA (1997).
48. G. John, R. Kohavi and K. Pfleger, Irrelevant features and the subset selection problem, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, Los Altos, CA, 1994) 121-129.
49. S. Judd, On the complexity of loading shallow neural networks, J. Complexity 4 (1988) 177-192.
50. L.P. Kaelbling, Learning in Embedded Systems (MIT Press, Cambridge, MA, 1993).
51. K. Kira and L.A. Rendell, The feature selection problem: Traditional methods and a new algorithm, in: Proceedings AAAI-92, San Jose, CA (MIT Press, Cambridge, MA, 1992) 129-134.
52. K. Kira and L.A. Rendell, A practical approach to feature selection, in: Proceedings 9th International Conference on Machine Learning, Aberdeen, Scotland (Morgan Kaufmann, Los Altos, CA, 1992).
53. J. Kittler, Une généralisation de quelques algorithms sous-optimaux de recherche d'ensembles d'attributs, in: Proceedings Congrés Reconnaissance des Formes et Traitement des Images (1978).
54. J. Kittler, Feature Selection and Extraction (Academic Press, New York, 1986) Chapter 3, 59-83.
55. R. Kohavi, Feature subset selection as search with probabilistic estimates, in: Proceedings AAAI Fall Symposium on Relevance, New Orleans, LA (1994) 122-126.
56. R. Kohavi, The power of decision tables, in: N. Lavrac and S. Wrobel, eds., Proceedings European Conference on Machine Learning, Lecture Notes in Artificial Intelligence, Vol. 914 (Springer, Berlin, 1995) 174-189.
57. R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in: C.S. Mellish, ed., Proceedings IJCAI-95, Montreal, Que. (Morgan Kaufmann, Los Altos, CA, 1995) 1137-1143.
58. R. Kohavi, Wrappers for performance enhancement and oblivious decision graphs, Ph.D. Thesis, Stanford University, Computer Science Department, STAN-CS-TR-95-1560 (1995); ftp:// starry.stanford.edu/pub/ronnyk/teza.ps.
59. R. Kohavi and B. Frasca, Useful feature subsets and rough set reducts, in: Proceedings 3rd International Workshop on Rough Sets and Soft Computing (1994) 310-317; also: in: Soft Computing by Lin and Wildberger.
60. R. Kohavi and G. John, Automatic parameter selection by minimizing estimated error, in: A. Prieditis and S. Russell, eds., Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, Los Altos, CA, 1995) 304-312.
61. R. Kohavi and D. Sommerfield, Feature subset selection using the wrapper model: overfitting and dynamic search space topology, in: Proceedings 1st International Conference on Knowledge Discovery and Data Mining (1995) 192-197.
62. R. Kohavi and D.H. Wolpert, Bias plus variance decomposition for zero-one loss functions, in: L. Saitta, ed., Proceedings 13th International Conference on Machine Learning, Bari, Italy (Morgan Kaufmann, Los Altos, CA, 1996) 275-283; available at: http://robotics.stanford.edu/users/ronnyk.
63. R. Kohavi, D. Sommerfield and J. Dougherty, Data mining using MLC++: A machine learning library in C++, in: Tools with Artificial Intelligence (IEEE Computer Society Press, Rockville, MD, 1996) 234-245; http://www.sgi.com/Technology/mlc.
64. I. Kononenko, Estimating attributes: analysis and extensions of Relief, in: F. Bergadano and L. De Raedt, eds., Proceedings European Conference on Machine Learning (1994).
65. I. Kononenko, On biases in estimating multi-valued attributes, in: C.S. Mellish, ed., Proceedings IJCAI-95, Montreal, Que. (Morgan Kaufmann, Los Altos, CA, 1995) 1034-1040.
66. J. Koza, Genetic Programming: On the Programming of Computers by Selection Means of Natural Selection (MIT Press, Cambridge, MA, 1992).
67. A. Krogh and J. Vedelsby, Neural network ensembles, cross validation, and active learning, in: Advances in Neural Information Processing Systems, Vol. 7 (MIT Press, Cambridge, MA, 1995).
68. S.W. Kwok and C. Carter, Multiple decision trees, in: R.D. Schachter, T.S. Levitt, L.N. Kanal and J.F. Lemmer, eds., Uncertainty in Artificial Intelligence (Elsevier, Amsterdam, 1990) 327-335.
69. P. Laarhoven and E. Aarts, Simulated annealing: Theory and Applications (Kluwer Academic Publishers, Dordrecht, 1987).
70. P. Langley, Selection of relevant features in machine learning, in: Proceedings AAAI Fall Symposium on Relevance, New Orleans, LA (1994) 140-144.
71. P. Langley and S. Sage, Sage, Induction of selective Bayesian classifiers, in: Proceedings 10th Conference on Uncertainty in Artificial Intelligence, Seattle, WA (Morgan Kaufmann, San Mateo, CA, 1994) 399-406.
72. P. Langley and S. Sage, Oblivious decision trees and abstract cases, in: Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, Seattle, WA (AAAI Press, 1994) 113-117.
73. P. Langley, W. Iba and K. Thompson, An analysis of Bayesian classifiers, in: Proceedings AAAI-94, Seattle, WA (AAAI Press and MIT Press, 1992) 223-228.
74. H. Linhart and W. Zucchini, Model Selection (Wiley, New York, 1986).
75. N. Littlestone and M.K. Warmuth, The weighted majority algorithm, Inform. and Comput. 108 (1994) 212-261.
76. p, Technometrics 15 (1973) 661-675.
77. T. Marill and D.M. Green, On the effectiveness of receptors in recognition systems, IEEE Trans. Inform. Theory 9 (1963) 11-17.
78. O. Maron and A.W. Moore, Hoeffding races: accelerating model selection search for classification and function approximation, in: Advances in Neural Information Processing Systems, Vol. 6 (Morgan Kaufmann, Los Altos, CA, 1994).
79. C.J. Merz and P.M. Murphy, UCI repository of machine learning databases (1996); http://www. ics.uci.edu/∼mlearn/MLRepository.html.
80. A.J. Miller, Selection of subsets of regression variables, J. Roy. Statist. Soc. A 147 (1984) 389-425.
81. A.J. Miller, Subset Selection in Regression (Chapman and Hall, London, 1990).
82. M.L. Minsky and S. Papert, Perceptrons: an Introduction to Computational Geometry (MIT Press, Cambridge, MA, expanded ed., 1988)
83. D. Mladenić, Automated model selection, in: ECML Workshop on Knowledge Level Modeling and Machine Learning (1995).
84. M. Modrzejewski, Feature selection using rough sets theory, in: P.B. Brazdil, ed., Proceedings European Conference on Machine Learning (Springer, Berlin, 1993) 213-226.
85. A.W. Moore and M.S. Lee, Efficient algorithms for minimizing cross validation error, in: W.W. Cohen and H. Hirsh, eds., Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, Los Altos, CA, 1994).
86. B.M.E. Moret, Decision trees and diagrams, ACM Comput. Surveys 14 (1982) 593-623.
87. S. Murthy and S. Salzberg, Lookahead and pathology in decision tree induction, in: C.S. Mellish, ed., Proceedings IJCAI-95, Montreal, Que. (Morgan Kaufmann, Los Altos, CA, 1995) 1025-1031.
88. M.P. Narendra and K. Fukunaga, A branch and bound algorithm for feature subset selection, IEEE Trans. Comput. 26 (1977) 917-922.
89. J. Neter, W. Wasserman and M.H. Kutner, Applied Linear Statistical Models (Irwin, Homewood, IL, 3rd ed., 1990).
90. Z. Pawlak, Rough Sets (Kluwer Academic Publishers, Dordrecht, 1991).
91. Z. Pawlak, Rough sets: present state and the future, Found. Comput. Decision Sci. 18 (1993) 157-166.
92. M.J. Pazzani, Searching for dependencies in Bayesian classifiers, in: D. Fisher and H. Lenz, eds., Proceedings 5th International Workshop on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, 1995.
93. M. Perrone, Improving regression estimation: averaging methods for variance reduction with extensions to general convex measure optimization, Ph.D. Thesis, Physics Department, Brown University, Providence, RI (1993).
94. G.M. Provan and M. Singh, Learning Bayesian networks using feature selection, in: D. Fisher and H. Lenz, eds., Proceedings 5th International Workshop on Artificial Intelligence and Statistics, Ft. Lauderdale, FL (1995) 450-456.
95. F.J. Provost, Policies for the selection of bias in inductive machine learning, Ph.D. Thesis, Computer Science Department, University of Pittsburgh, Rept. No. 92-34 (1992).
96. F.J. Provost and B.G. Buchanan, Inductive policy: the pragmatics of bias selection, Machine Learning 20 (1995) 35-61.
97. J.R. Quinlan, Induction of decision trees, Machine Learning 1 (1986) 81-106;
98. reprinted in: J.W. Shavlik and T.G. Dietterich, eds., Readings in Machine Learning (Morgan Kaufmann, San Mateo, CA, 1986).
99. J.R. Quinlan, C4.5: Programs for Machine Learning (Morgan Kaufmann, San Mateo, CA, 1993).
100. J.R. Quinlan, Oversearching and layered search in empirical learning, in: C.S. Mellish, ed., Proceedings IJCAI-95, Montreal, Que. (Morgan Kaufmann, Los Altos, CA, 1995) 1019-1024.
101. L. Rendell and R. Seshu, Learning hard concepts through constructive induction: Framework and rationale, Comput. Intell. 6 (1990) 247-270.
102. F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain, Psychological Review 65 (1958) 386-408.
103. S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach (Prentice Hall, Englewood Cliffs, NJ, 1995).
104. C. Schaffer, Selecting a classification method by cross-validation, Machine Learning 13 (1993) 135-143.
105. R.E. Schapire, The strength of weak learnability, Machine Learning 5 (1990) 197-227.
106. W. Siedlecki and J. Sklansky, On automatic feature selection, Internat. J. Pattern Recognition and Artificial Intelligence 2 (1988) 197-220.
107. M. Singh and G.M. Provan, A comparison of induction algorithms for selective and non-selective Bayesian classifiers, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Mateo, CA, 1995) 497-505.
108. D.B. Skalak, Prototype and feature selection by sampling and random mutation hill climbing algorithms, in: W.W. Cohen and H. Hirsh, eds., Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, Los Altos, CA, 1994).
109. W.N. Street, O.L. Mangasarian and W.H. Wolberg, An inductive learning approach to prognostic prediction, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Mateo, CA, 1995).
110. C. Taylor, D. Michie and D. Spiegelhalter, Machine Learning, Neural and Statistical Classification (Paramount Publishing International, 1994).
111. Thrun et al., The Monk's problems: a performance comparison of different learning algorithms, Tech. Rept. CMU-CS-91-197, Carnegie Mellon University, Pittsburgh, PA (1991).
112. P.D. Turney, Exploiting context when learning to classify, in: P.B. Brazdil, ed., Proceedings European Conference on Machine Learning 〈ECML〉 (1993) 402-407.
113. P.D. Turney, The identification of context-sensitive features, a formal definition of context for concept learning, in: M. Kubat and G. Widmer, eds., Proceedings Workshop on Learning in Context-Sensitive Domains (1996) 53-59; also available as: National Research Council of Canada Tech. Rept. #39222.
114. P.E. Utgoff, An improved algorithm for incremental induction of decision trees, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, Los Altos, CA, 1994) 318-325.
115. P.E. Utgoff, Decision tree induction based on efficient tree restructuring, Tech. Rept. 05-18, University of Massachusetts, Amherst, MA (1995).
116. H. Vafai and K. De Jong, Genetic algorithms as a tool for feature selection in machine learning, in: Proceedings 4th International Conference on Tools with Artificial Intelligence (IEEE Computer Society Press, Rockville, MD, 1992) 200-203.
117. H. Vafai and K. De Jong, Robust feature selection algorithms, in Proceedings 5th International Conference on Tools with Artificial Intelligence (IEEE Computer Society Press, Rockville, MD, 1993) 356-363.
118. D.H. Wolpert, On the connection between in-sample testing and generalization error, Complex Systems 6 (1992) 47-94.
119. D.H. Wolpert, Stacked generalization, Neural Networks 5 (1992) 241-259.
120. L. Xu, P. Yan and T. Chang, Best first strategy for feature selection, in: Proceedings 9th International Conference on Pattern Recognition (IEEE Computer Society Press, Rockville, MD, 1989) 706-708.
121. D. Yan and H. Mukai, Stochastic discrete optimization, SIAM J. Control and Optimization 30 (1992) 594-612.
122. B. Yu and B. Yuan, A more efficient branch and bound algorithm for feature selection, Pattern Recognition 26 (1993) 883-889.
123. W. Ziarko, The discovery, analysis and representation of data dependencies in databases, in: G. Piatetsky-Shapiro and W. Frawley, eds., Knowledge Discovery in Databases (MIT Press, Cambridge, MA, 1991).