Selection of relevant features and examples in machine learning

Artificial Intelligence

Volumn 97, Issue 1-2, 1997, Pages 245-271

  Blum, Avrim L ^a   Langley, Pat ^b,c  

^a Carnegie Mellon University ^*  (United States)

^b Institute for the Study of Learning and Expertise   (United States)

^c DaimlerChrysler Research and Technology North America Inc   (United States)

Author keywords

Machine learning; Relevant examples; Relevant features

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DATA HANDLING; DATA STRUCTURES;

RELEVANCE REASONING;

LEARNING SYSTEMS;

EID: 0031334221     PISSN: 00043702     EISSN: None     Source Type: Journal    
DOI: 10.1016/s0004-3702(97)00063-5     Document Type: Article

References (103)

1. D. Aha, A study of instance-based algorithms for supervised learning tasks: mathematical, empirical and psychological evaluations, Doctoral Dissertation, Department of Information and Computer Science, University of California, Irvine, CA (1990).
2. D.W. Aha and R.L. Bankert, A comparative evaluation of sequential feature selection algorithms, in: D. Fisher and J.-H. Lenz, eds., Artificial Intelligence and Statistics V (Springer, New York, 1996).
3. H. Almuallim and T.G. Dietterich, Learning with many irrelevant features, in: Proceedings AAAI-91, Anaheim, CA (AAAI Press, 1991) 547-552.
4. D. Angluin, Learning regular sets from queries and counterexamples, Inform. and Comput. 75 (1987) 87-106.
5. D. Angluin, L. Hellerstein and M. Karpinski, Learning read-once formulas with queries, J. ACM 40 (1993) 185-210.
6. R. Armstrong, D. Freitag, T. Joachims and T. Mitchell, Webwatcher: a learning apprentice for the World Wide Web, in: Proceedings AAAI Spring Symposium on Information Gathering from Heterogeneous Distributed Environments (1993).
7. S. Baluja and D. Pomerleau, Dynamic relevance: vision-based focus of attention using artificial neural networks (Technical Note), Artificial Intelligence 97 (1997) 381-395 (this issue).
8. A. Blum, Learning Boolean functions in an infinite attribute space, Machine Learning 9 (1992) 373-386.
9. A. Blum, Empirical support for WINNOW and weighted-majority based algorithms: results on a calendar scheduling domain, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Mateo, CA, 1995) 64-72.
10. A. Blum, M. Furst, J. Jackson, M. Kearns, Y. Mansour and S. Rudic, Weakly learning DNF and characterizing statistical query learning using Fourier analysis, in: Proceedings 26th Annual ACM Symposium on Theory of Computing, Montreal, Que. (1994) 253-262.
11. A. Blum, L. Hellerstein and N. Littlestone, Learning in the presence of finitely or infinitely many irrelevant attributes, J. Comput. System Sci. 50 (1995) 32-40.
12. A. Blum and R. Kannan, Learning an intersection of k halfspaces over a uniform distribution, in: Proceedings 34th Annual IEEE Symposium on Foundations of Computer Science, Palo Alto, CA (IEEE, 1993) 312-320.
13. A. Blumer, A. Ehrenfeucht, D. Haussler and M.K. Warmuth, Occam's razor, Inform. Process. Lett. 24 (1987) 377-380.
14. A. Blumer, A. Ehrenfeucht, D. Haussler and M.K. Warmuth, Learnability and the Vapnik-Chervonenkis dimension, J. ACM 36 (1989) 929-965.
15. L. Breiman, J.H. Friedman, R.A. Olshen and C.J. Stone, Classification and Regression Trees (Wadsworth, Belmont, CA 1984).
16. N.H. Bshouty, Exact learning via the monotone theory, in: Proceedings IEEE Symposium on Foundations of Computer Science, Palo Alto, CA (IEEE, 1993) 302-311.
17. C. Cardie, Using decision trees to improve case-based learning, in: Proceedings 10th International Conference on Machine Learning, Amherst, MA (Morgan Kaufmann, San Mateo, CA, 1993) 25-32.
18. R.A. Caruana and D. Freitag, Greedy attribute selection, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 28-36.
19. R.A. Caruana and D. Freitag, How useful is relevance?, in: Working Notes of the AAAI Fall Symposium on Relevance, New Orleans, LA (AAAI Press, 1994) 25-29.
20. J. Catlett, Peepholing: choosing attributes efficiently for megainduction, in: Proceedings 9th International Conference on Machine Learning, Aberdeen, Scotland (Morgan Kaufmann, San Mateo, CA, 1992) 49-54.
21. N. Cesa-Bianchi, Y. Freund, D.P. Helmbold, D. Haussler, R.E. Schapire and M.K. Warmuth, How to use expert advice, Proceedings Annual ACM Symposium on the Theory of Computing (1993) 382-391.
22. P. Clark and T. Niblett, The CN2 induction algorithm, Machine Learning 3 (1989) 261-284.
23. D.A. Cohn, Z. Ghahramani and M.I. Jordan, Active learning with statistical models, J. Artif. Intell. Research 4 (1996) 129-145.
24. P. Comon, Independent component analysis: a new concept, Signal Process. 36 (1994) 287-314.
25. T.M. Cover and P.E. Hart, Nearest neighbor pattern classification, IEEE Trans. Inform. Theory 13 (1967) 21-27.
26. W. Daelemans, S. Gillis and G. Durieux, The acquisition of stress: a data-oriented approach, Comput. Linguistics 20 (3) (1994) 421-451.
27. P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach (Prentice-Hall, Englewood Cliffs, NJ, 1982).
28. A. Dhagat and L. Hellerstein, PAC learning with irrelevant attributes, in: Proceedings IEEE Symposium on Foundations of Computer Science (IEEE, 1994) 64-74.
29. J. Doak, An evaluation of feature-selection methods and their application to computer security, Tech. Rept. CSE-92-18, Department of Computer Science, University of California at Davis (1992).
30. H. Drucker, R. Schapire and P. Simard, Improving performance in neural networks using a boosting algorithm, in: Advances in Neural Information Processing Systems, Vol. 4 (Morgan Kaufmann, San Mateo, CA, 1992).
31. H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun and V. Vapnik, Boosting and other machine learning algorithms, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 53-61.
32. M. Dyer, A. Frieze and R. Kannan, A random polynomial time algorithm for approximating the volume of convex bodies, in: Proceedings Annual ACM Symposium on the Theory of Computing (1989) 375-381.
33. Y. Freund, Boosting a weak learning algorithm by majority, in: Proceedings 3rd Annual Workshop on Computational Learning Theory, San Francisco, CA (Morgan Kaufmann, San Mateo, CA, 1990) 202-216.
34. Y. Freund, An improved boosting algorithm and its implications on learning complexity, in: Proceedings 5th Annual ACM Workshop on Computational Learning Theory, Pittsburgh, PA (ACM Press, 1992) 391-398.
35. M. Garey and D. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness (W.H. Freeman, San Francisco, CA, 1979).
36. Y. Gil, Efficient domain-independent experimentation, in: Proceedings 10th International Conference on Machine Learning, Amherst, MA, (Morgan Kaufmann, San Mateo, CA, 1993) 128-134.
37. R. Greiner, A.J. Grove and A. Kogan, Knowing what doesn't matter: exploiting the omission of irrelevant data, Artificial Intelligence 97 (1997) 345-380 (this issue).
38. K.P. Gross, Concept acquisition through attribute evolution and experiment selection, Doctoral Dissertation, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA (1991).
39. D. Haussler, Quantifying the inductive bias in concept learning, in: Proceedings AAAI-86, Philadelphia, PA (AAAI Press, 1986) 485-489.
40. R. Holte, Very simple classification rules perform well on most commonly used domains, Machine Learning 11 (1993) 63-91.
41. J. Jackson, An efficient membership-query algorithm for learning DNF with respect to the uniform distribution, in: Proceedings IEEE Symposium on Foundations of Computer Science (IEEE, 1994).
42. G.H. John, R. Kohavi and K. Pfleger, Irrelevant features and the subset selection problem, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 121-129.
43. G.H. John and P. Langley, Static vs. dynamic sampling for data mining, in: Proceedings 2nd International Conference of Knowledge Discovery and Data Mining, Portland, OR (AAAI Press, 1996) 367-370.
44. I.T. Jolliffe, Principal Component Analysis (Springer, New York, 1986).
45. D.S. Johnson, Approximation algorithms for combinatorial problems, J. Comput. System Sci. 9 (1974) 256-278.
46. M.J. Kearns and U.V. Vazirani, An Introduction to Computational Learning Theory (MIT Press, Cambridge, MA, 1994).
47. K. Kira and L. Rendell, A practical approach to feature selection, in: Proceedings 9th International Conference on Machine Learning, Aberdeen, Scotland (Morgan Kaufmann, San Mateo, CA, 1992) 249-256.
48. J. Kivinen and M.K. Warmuth, Additive versus exponentiated gradient updates for linear prediction, in: Proceedings 27th Annual ACM Symposium on Theory of Computing, New York (ACM Press, 1995) 209-218.
49. J. Kivinen, M.K. Warmuth and P. Auer, The Perceptron algorithm versus Winnow: linear versus logarithmic mistake bounds when few input variables are relevant ( Technical Note), Artificial Intelligence 97 (1997) 325-343 (this issue).
50. R. Kohavi, The power of decision tables, in: Proceedings 8th European Conference on Machine Learning (1995).
51. R. Kohavi and G.H. John, Wrappers for feature subset selection. Artificial Intelligence 97 (1997) 273-324 (this issue).
52. R. Kohavi, P. Langley and Y. Yun, The utility of feature weighting in nearest-neighbor algorithms, in: Proceedings 9th European Conference on Machine Learning, Prague (Springer, Berlin, 1997).
53. B. Knobe and K. Knobe, A method for inferring context-free grammars, Inform. and Control 31 (1977) 129-146.
54. D. Koller and M. Sahami, Toward optimal feature selection, in: Proceedings 13th International Conference on Machine Learning, Bari, Italy (Morgan Kaufmann, San Mateo, CA, 1996).
55. I. Kononenko, Estimating attributes: analysis and extensions of RELIEF, in: Proceedings 7th European Conference on Machine Learning (1994).
56. M. Kubat, D. Flotzinger and G. Pfurtscheller, Discovering patterns in EEG signals: comparative study of a few methods, in: Proceedings 1993 European Conference on Machine Learning, Vienna (Springer, Berlin, 1993) 367-371.
57. D. Kulkarni and H.A. Simon, Experimentation in machine discovery, in: J. Shrager and P. Langley, eds., Computational Models of Scientific Discovery and Theory Formation (Morgan Kaufmann, San Mateo, CA, 1990).
58. P. Langley and W. Iba, Average-case analysis of a nearest neighbor algorithm, in: Proceedings IJCAI-93, Chambery, France (1993) 889-894.
59. P. Langley and S. Sage, Oblivious decision trees and abstract cases, in: Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, Seattle, WA (AAAI Press, 1994) 113-117.
60. P. Langley and S. Sage, Induction of selective Bayesian classifiers, in: Proceedings 10th Conference on Uncertainty in Artificial Intelligence, Seattle, WA (Morgan Kaufmann, San Mateo, CA, 1994) 399-406.
61. P. Langley and S. Sage, Scaling to domains with many irrelevant features, in: R. Greiner, ed., Computational Learning Theory and Natural Learning Systems, Vol. 4 (MIT Press, Cambridge, MA, 1997).
62. D.D. Lewis, Representation and learning in information retrieval, Doctoral Dissertation, Tech. Rept. UM-CS-1991-093, Department of Computer Science, University of Massachusetts, Amherst, MA (1992).
63. D.D. Lewis, Feature selection and feature extraction for text categorization, in: Proceedings Speech and Natural Language Workshop, San Francisco (Morgan Kaufmann, San Mateo, CA, 1992) 212-217.
64. D.D. Lewis and J. Catlett, Heterogeneous uncertainty sampling, in: Proceedings llth International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 148-156.
65. L.J. Lin, Self-improving reactive agents based on reinforcement learning, planning and teaching, Machine Learning 8 (1992) 293-321.
66. N. Littlestone, Learning quickly when irrelevant attributes abound: a new linear threshold algorithm, Machine Learning 2 (1988) 285-318.
67. N. Littlestone, P.M. Long and M.K. Warmuth, On-line learning of linear functions, in: Proceedings 23rd Annual ACM Symposium on Theory of Computing, New Orleans, LA (ACM Press, 1991) 465-475.
68. N. Littlestone and C. Mesterharm, An apobayesian relative of WINNOW, in: M.C. Mozer, M.I. Jordan and T. Petsche, eds., Advances in Neural Information Processing Systems, Vol. 9 (MIT Press, Cambridge, MA, 1997).
69. N. Littlestone and M.K. Warmuth, The weighted majority algorithm, Inform. and Comput. 108 (1994) 212-261.
70. L. Lovász and M. Simonovits, On the randomized complexity of volume and diameter, in: Proceedings IEEE Symposium on Foundations of Computer Science (IEEE, 1992) 482-491.
71. C. Lund and M. Yannakakis, On the hardness of approximating minimization problems, in: Proceedings Annual ACM Symposium on the Theory of Computing (1993) 286-293).
72. C.J. Matheus and L.A. Rendell, Constructive induction on decision trees, in: Proceedings IJCAI-89, Detroit, MI (Morgan Kaufmann, San Mateo, CA, 1989) 645-650.
73. R.S. Michalski, Pattern recognition as rule-guided inductive inference, IEEE Trans. Pattern Anal. Machine Intell. 2 (1980) 349-361.
74. M. Minsky and S. Papert, Perceptrons: An Introduction to Computational Geometry (MIT Press, Cambridge, MA, 1969).
75. T.M. Mitchell, Generalization as search, Artificial Intelligence 18 (1982) 203-226; reprinted in: J.W. Shavlik and T.G. Dietterich, eds., Readings in Machine Learning (Morgan Kaufmann, San Mateo, CA, 1990).
76. T.M. Mitchell, Generalization as search, Artificial Intelligence 18 (1982) 203-226; reprinted in: J.W. Shavlik and T.G. Dietterich, eds., Readings in Machine Learning (Morgan Kaufmann, San Mateo, CA, 1990).
77. A.W. Moore and M.S. Lee, Efficient algorithms for minimizing cross validation error, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 190-198.
78. S.W. Norton, Generating better decision trees, in: Proceedings IJCAI-89, Detroit, MI (Morgan Kaufmann, San Mateo, CA, 1989) 800-805.
79. M.J. Pazzani and W. Sarrett, A framework for the average case analysis of conjunctive learning algorithms, Machine Learning 9 (1992) 349-372.
80. G. Pagallo and D. Haussler, Boolean feature discovery in empirical learning, Machine Learning 5 (1990) 71-99.
81. J.R. Quinlan, Learning efficient classification procedures and their application to chess end games, in: R.S. Michalski, J.G. Carbonell and T.M. Mitchell, eds., Machine Learning: An Artificial Intelligence Approach (Morgan Kaufmann, San Mateo, CA, 1983).
82. J.R. Quinlan, C4.5: Programs for Machine Learning (Morgan Kaufmann, San Mateo, CA, 1993).
83. S. Rajamoney, A computational approach to theory revision, in: J. Shrager and P. Langley, eds., Computational Models of Scientific Discovery and Theory Formation (Morgan Kaufmann, San Mateo, CA, 1990).
84. R.L. Rivest and R.E. Schapire, Inference of finite automata using homing sequences, Inform. and Comput. 103 (1993) 299-347.
85. D.E. Rumelhart, G. Hinton and R.J. Williams, Learning internal representations by error propagation, in: D.E. Rumelhart and J.L. McClelland, and the PDP Research Group, eds., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1 (MIT Press, Cambridge, MA, 1986).
86. C Sammut and R.B. Banerji, Learning concepts by asking questions, in: R.S. Michalski, J.G. Carbonell and T.M. Mitchell, eds., Machine Learning: An Artificial Intelligence Approach, Vol. 2 (Morgan Kaufmann, San Mateo, CA, 1986).
87. R.E. Schapire, The strength of weak learnability, Machine Learning 5 (1990) 197-227.
88. J.C. Schlimmer, Efficiently inducing determinations: a complete and efficient search algorithm that uses optimal pruning, in: Proceedings 10th International Conference on Machine Learning, Amherst, MA (Morgan Kaufmann, San Mateo, CA, 1993) 284-290.
89. P.D. Scott and S. Markovitz, Representation generation in an exploratory learning system, in: D.H. Fisher, M.J. Pazzani and P. Langley, eds., Concept Formation: Knowledge and Experience in Unsupervised Learning (Morgan Kaufmann, San Mateo, CA, 1991).
90. H.S. Seung, M. Opper and H. Sompolinsky, Query by committee, Proceedings 5th Annual Workshop on Computational Learning Theory, Pittsburgh, PA (ACM Press, New York, 1992) 287-294.
91. W.M. Shen and H.A. Simon, Rule creation and rule learning through environmental exploration, in: Proceedings IJCAI-89, Detroit, MI (Morgan Kaufmann, San Mateo, CA, 1989) 675-680.
92. A. Sinclair and M. Jerrum, Approximate counting, uniform generation and rapidly mixing Markov chains, Inform. and Comput. 82 (1989) 93-133.
93. M. Singh and G.M. Provan, A comparison of induction algorithms for selective and non-selective Bayesian classifiers, in: Proceedings 12th International Conference on Machine Learning, Lake Tahoe, CA (Morgan Kaufmann, San Mateo, CA, 1995) 497-505.
94. M. Singh and G.M. Provan, Efficient learning of selective Bayesian network classifiers, in: Proceedings 13th International Conference on Machine Learning, Bari, Italy (Morgan Kaufmann, San Mateo, CA, 1996).
95. D.B. Skalak, Prototype and feature selection by sampling and random mutation hill-climbing algorithms, in: Proceedings 11th International Conference on Machine Learning, New Brunswick, NJ (Morgan Kaufmann, San Mateo, CA, 1994) 293-301.
96. C.W. Stanfill, Memory-based reasoning applied to English pronunciation, in: Proceedings AAAI-87, Seattle, WA (AAAI Press, 1987) 577-581.
97. K.M. Ting, Discretization of continuous-valued attributes and instance-based learning, Tech. Rept. No. 491, Basser Department of Computer Science, University of Sydney (1994).
98. T. Townsend-Weber and D. Kibler, Instance-based prediction of continuous values, in: Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, Seattle, WA (AAAI Press, 1994) 30-35.
99. K. Verbeurgt, Learning DNF under the uniform distribution in polynomial time, in: Proceedings 3rd Annual Workshop on Computational Learning Theory, San Francisco, CA (Morgan Kaufmann, San Mateo, CA, 1990) 314-325.
100. S.A. Vere, Induction of concepts in the predicate calculus, in: Proceedings IJCAI-75, Tbilisi, Georgia (Morgan Kaufmann, San Mateo, CA, 1975) 281-287.
101. V. Vovk, Aggregating strategies, in: Proceedings 3rd Annual Workshop on Computational Learning Theory, San Francisco, CA (Morgan Kaufmann, San Mateo, CA, 1990) 371-383.
102. B. Widrow and M.E. Hoff, Adaptive switching circuits, in: IRE WESCON Convention Record (1960) 96-104.
103. P.H. Winston, Learning structural descriptions from examples, in: P.H. Winston, ed., The Psychology of Computer Vision (McGraw-Hill, New York, 1975).