A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms

Artificial Intelligence Review

Volumn 11, Issue 1-5, 1997, Pages 273-314

  Wettschereck, Dietrich ^a   Aha, David W ^b   Mohri, Takao ^c  

^a FRAUNHOFER INSTITUTE FOR APPLIED INFORMATION TECHNOLOGY FIT   (Germany)

^b NAVAL RESEARCH LABORATORY   (United States)

^c UNIVERSITY OF TOKYO   (Japan)

Author keywords

Comparison; Feature weights; k nearest neighbor; Lazy learning

Indexed keywords

FEATURE EXTRACTION; FUNCTIONS; LEARNING SYSTEMS; SENSITIVITY ANALYSIS;

DISTANCE FUNCTION; FEATURE WEIGHTS; LAZY LEARNING; NEAREST NEIGHBOR (NN);

LEARNING ALGORITHMS;

EID: 0031073477     PISSN: 02692821     EISSN: None     Source Type: Journal    
DOI: 10.1007/978-94-017-2053-3_11     Document Type: Review

References (108)

1. Aha, D. W. (1991). Incremental constructive induction: An instance-based approach. In Proceedings of the Eighth International Workshop on Machine Learning, pp. 117-121. Evanston, IL: Morgan Kaufmann.
2. Aha, D. W. (1992). Tolerating noisy, irrelevant, and novel attributes in instance-based learning algorithms. International Journal of Man-Machine Studies 36: 267-287.
3. Aha, D. W. & Bankert, R. L. (1994). Feature selection for case-based classification of cloud types: An empirical comparison. In D. W. Aha (ed.) Case-Based Reasoning: Papers from the 1994 Workshop (Technical Report WS-94-01). Menlo Park, CA: AAAI Press.
4. Aha, D. W. & Goldstone, R. L. (1992). Concept learning and flexible weighting. In Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society, pp. 534-539. Bloomington, IN: Lawrence Erlbaum.
5. Aha, D. W., Kibler, D. & Albert, M. K. (1991). Instance-based learning algorithms. Machine Learning, 6: 37-66.
6. Ashley, K. D. & Rissland, E. L. (1988). Waiting on weighting: A symbolic least commitment approach. In Proceedings of the Seventh National Conference on Artificial Intelligence, pp. 239-244. St. Paul, MN: Morgan Kaufmann.
7. Atkeson, C. (1989). Using local models to control movement. In D. S. Touretzky (ed.), Advances in Neural Information Processing Systems 2. San Mateo, CA: Morgan Kaufmann.
8. Atkeson, C., Moore, A. & Schaal, S. (1996a). Locally weighted learning. Artificial Intelligence Review, this issue.
9. Atkeson, C., Moore, A. & Schaal, S. (1996b). Locally weighted learning for control. Artificial Intelligence Review, this issue.
10. Bakiri, G. (1991). Converting English text to speech: A machine learning approach. Doctoral dissertation, Department of Computer Science, Oregon State University, Corvallis, OR.
11. Bareiss, R. (1989). The experimental evaluation of a case-based learning apprentice. In Proceedings of a Case-Based Reasoning Workshop, pp. 162-167. Pensacola Beach, FL: Morgan Kaufmann.
12. Battiti, R. (1994). Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks 5: 537-550.
13. Biberman, Y. (1994). A context similarity measure. In Proceedings of the European Conference on Machine Learning, pp. 49-63. Catania, Italy: Springer-Verlag.
14. Bottou, L. & Vapnik, V. (1992). Local learning algorithms. Neural Computation 4: 888-900.
15. Bounds, D., Lloyd, P. & Mathew, B. (1990). A comparison of neural network and other pattern recognition approaches to the diagnosis of low back disorders. Neural Networks 3: 583-591.
16. Broomhead, D. S. & Lowe, D. (1988). Multivariable functional interpolation and adaptive networks. Complex Systems 2: 321-355.
17. Cain, T., Pazzani, M. J. & Silverstein, G. (1991). Using domain knowledge to influence similarity judgement. In Proceedings of the Case-Based Reasoning Workshop, pp. 191-202. Washington, DC: Morgan Kaufmann.
18. Cardie, C. (1993). Using decision trees to improve case-based learning. In Proceedings of the Tenth International Conference on Machine Learning, pp. 25-32. Amherst, MA: Morgan Kaufmann.
19. Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, J.H. & Rosen, D.B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks 3: 693-713.
20. Cleveland, W. S. & Loader, C. (1994). Computational methods for local regression (Technical Report 11). Murray Hill, NJ: AT&T Bell Laboratories, Statistics Department. Available by FTP from netlib.att.com in /netlib/att/stat/doc/94/11.ps.
21. Connell, M. E. & Utgoff, P. E. (1987). Learning to control a dynamic physical system. In Proceedings of the Sixth National Conference on Artificial Intelligence, pp. 456-460. Seattle, WA: Morgan Kaufmann.
22. Cost, S. & Salzberg, S. (1993). A weighted nearest neighbor algorithm for learning with symbolic features. Machine Learning 10: 57-78.
23. Cover, T. M. & Hart, P. E. (1967). Nearest neighbor pattern classification. Institute of Electrical and Electronics Engineers Transactions on Information Theory 13: 21-27.
24. Cover, T. M. & Thomas, J. (1991). Elements of Information Theory. New York: John Wiley and Sons.
25. Cover, T. M. & van Campenhout, J. M. (1977). On the possible orderings in the measurement selection problem. IEEE Transactions on Systems, Man, and Cybernetics 7: 657-661.
26. Creecy, R. H., Masand, B. M., Smith, S. J. & Waltz, D. L. (1992). Trading MIPS and memory for knowledge engineering. Communications of the ACM 35: 48-64.
27. Daelemans, W., Gills, S. & Durieux, G. (1993). Learnability and markedness in data-driven acquisition of stress (Technical Report 43). Tilburg, Netherlands: Tilburg University, Institute for Language Technology and Artificial Intelligence.
28. Daelemans, W., van den Bosch, A. (1992). Generalization performance of backpropagation learning on a syllabification task. In Proceedings of TWLT3: Connectionism and Natural Language Processing, pp. 27-37. Enschede, The Netherlands: Unpublished.
29. Dasarathy, B. V. (Ed.). (1991). Nearest neighbor(NN) norms: NN pattern classification techniques. Los Alamitos, CA: IEEE Computer Society Press.
30. Doak, J. (1992). An evaluation of feature selection methods and their application to computer security (Technical Report CSE-92-18). Davis, CA: University of California, Department of Computer Science.
31. Domingos, P. (1996). Context-sensitive feature selection for lazy learners. Artificial Intelligence Review, this issue.
32. Duda, R. O. & Hart, P. E. (1973). Pattern classification and scene analysis. New York, NY: Wiley.
33. Dudani, S. (1975). The distance-weighted k-nearest neighbor rule. IEEE Transactions on Systems, Man, and Cybernetics 6: 325-327.
34. Fayyad, U. M., & Irani, K. B. (1993). Multi-interval discretization of continuous-valued attributes for classification learning. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp. 1022-1029. Chambery, France: Morgan Kaufmann.
35. Friedman, J. H. (1994). Flexible metric nearest neighbor classification. Unpublished manuscript available by anonymous FTP from playfair.stanford.edu (see pub/friedman/ README).
36. Fu, K. S. (1968). Sequential methods in pattern recognition and machine learning. New York: Academic Press.
37. Fukunaga, K. & Flick, T. (1982). A parametrically-defined nearest neighbor distance measure. Pattern Recognition Letters 1: 3-5.
38. Fukunaga, K. & Flick, T. (1984). An optimal global nearest neighbor metric. IEEE Transactions on Pattern Analysis and Machine Intelligence 6: 314-318.
39. Gorman, R. & Sejnowski, T. (1988). Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks 1: 75-89.
40. Hastie, T. J. & Tibshirani, R. J. (1994). Discriminant Adaptive Nearest Neighbor Classification. Unpublished manuscript available by anonymous FTP from playfair.stanford.edu as /pub/hastie/dann.ps.Z.
41. Hayashi, C. (1952). On the prediction of phenomena from qualitative data and the quantification of qualitative data from the mathematical-statistical point of view. Annals of the Institute of Statistical Mathematics 3: 69-98.
42. John, G., Kohavi, R. & Pfleger, K. (1994). Irrelevant features and the subset selection problem. In Proceedings of the Eleventh International Machine Learning Conference, pp. 121-129. New Brunswick, NJ: Morgan Kaufmann.
43. Kawaguchi, M. (1978). Introduction to Multivariate Analysis II (in Japanese). Morikita-Shuppan.
44. Kelly, J. D., Jr. & Davis, L. (1991). A hybrid genetic algorithm for classification. In Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, pp. 645-650. Sydney, Australia: Morgan Kaufmann.
45. Kibler, D. & Aha, D. W. (1987). Learning representative exemplars of concepts: An initial case study. In Proceedings of the Fourth International Workshop on Machine Learning, pp. 24-30. Irvine, CA: Morgan Kaufmann.
46. Kira, K. & Rendell, L. A. (1992). A practical approach to feature selection. In Proceedings of the Ninth International Conference on Machine Learning, pp. 249-256. Aberdeen, Scotland: Morgan Kaufmann.
47. Kohavi, R., Langley, P. & Yun, Y. (1995). Heuristic search for feature weights in instance-based learning. Manuscript submitted for publication.
48. Kohonen, T., Barna, G. & Chrisley, R. (1988). Statistical pattern recognition with neural networks. In Proceedings of the International Joint Conference on Neural Networks, pp. 61-88. IEEE Press.
49. Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE 78: 1464-1480.
50. Kolodner, J. (1993). Case-based reasoning. San Mateo, CA: Morgan Kaufmann.
51. Kononenko, I. (1994). Estimating attributes: Analysis and extensions of RELIEF. In Proceedings of the 1994 European Conference on Machine Learning, pp. 171-182. Catania, Italy: Springer Verlag.
52. Kruschke, J. K. (1992). ALCOVE: An exemplar-based connectionist model of category learning. Psychological Review 99: 22-44.
53. Kshirsager, A. (1972). Multivariate Analysis. New York: Dekker.
54. Langley, P. & Iba, W. (1993). Average-case analysis of a nearest neighbor algorithm. Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pp. 889-894. Chambery, France: Morgan Kaufmann.
55. Langley, P. & Sage, S. (1994). Oblivious decision trees and abstract cases. In D. W. Aha (ed.), Case-Based Reasoning: Papers from the 1994 Workshop (Technical Report WS-94-01). Menlo Park, CA: AAAI Press.
56. Ling, X. C. & Wang, H. (1996). Towards optimal weights setting for the 1-nearest neighbour learning algorithm. Artificial Intelligence Review, this issue.
57. Lowe, D. (1995). Similarity metric learning for a variable-kernel classifier. Neural Computation 7: 72-85.
58. Lucassen, J. & Mercer, R. (1984). An information theoretic approach to the automatic determination of phonemic base forms. In Proceedings of the International Conference on Acoustics Speech Signal Processing (42.5.1-42.5.4).
59. Luce, R. D. (1963). Detection and recognition. In R. D. Luce, R.R. Bush & E. Galanger (eds.), Handbook of mathematical psychology. New York, NY: Wiley.
60. McGill, W. (1955). Multivariate information transmission. IEEE Transactions on Information Theory 1: 93-111.
61. Medin, D. L. & Schaffer, M. M. (1978). Context theory of classification learning. Psychological Review 85: 207-238.
62. Michie, D., Spiegelhalter, D. J. & Taylor, C. C. (Eds.) (1994). Machine learning, neural and statistical classification. London: Prentice Hall.
63. Mitchell, T. M. (1990). The need for biases in learning generalizations. In J. W. Shavlik & T. G. Dietterich (eds.), Readings in machine learning. San Mateo, CA: Morgan Kaufmann.
64. Mitchell, T., Keller, R. & Kedar-Cabelli, S. (1986). Explanation-based learning: A unifying view. Machine Learning 1: 47-80.
65. Mohri, T., Nakamura, M. & Tanaka, H. (1993). Weather forecasting using memory-based reasoning. In Second International Workshop on Parallel Processing for Artificial Intelligence, pp. 40-45.
66. Mohri, T. & Tanaka, H. (1994). An optimal weighting criterion of case indexing for both numeric and symbolic attributes. In D. W. Aha (ed.), Case-Based Reasoning: Papers from the 1994 Workshop (Technical Report WS-94-01). Menlo Park, CA: AAAI Press.
67. Mohri, T. & Tanaka, H. (1995). Comparison between attribute weighting methods in memory-based reasoning and multivariate analysis. Manuscript submitted for publication.
68. Moore, A. W. & Lee, M. S. (1994). Efficient algorithms for minimizing cross validation error. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 190-198. New Brunswick, NJ: Morgan Kaufmann.
69. Mucciardi, A. N. & Gose, E. E. (1971). A comparison of seven techniques for choosing subsets of pattern recognition properties. IEEE Transaction on Computers 20: 1023-1031.
70. Murphy, P. (1995). UCI Repository of machine learning databases [Machine-readable data repository @ics.uci.edu]. Irvine, CA: University of California, Department of Information and Computer Science.
71. Myles, J. & Hand, D. (1990). The multi-class metric problem in nearest neighbor discrimination rules. Pattern Recognition 23: 1291-1297.
72. Nadaraya, E. A. (1964), On estimating regression. Theory of Probability and its Applications 9: 141-142.
73. Poggio, T. & Girosi, F. (1990). Regularization algorithms for learning that are equivalent to multilayer networks. Science 247: 978-982.
74. Porter, B. W., Bareiss, R. & Holte, R. C. (1990). Knowledge acquisition and heuristic classification in weak-theory domains. Artificial Intelligence 45: 229-263.
75. Press, W. H., Teukolsky, S. A., Vetterling, W. T. & Flannery, B. P. (1992). Numerical Recipes in C. Cambridge, UK: Cambridge University Press.
76. Quinlan, J. R. (1986). Induction of decision trees. Machine Learning 1: 81-106.
77. Ricci, F. & Avesani, P. (1995). Learning a local similarity metric for case-based reasoning. In Proceedings of the First International Conference on Case-Based Reasoning, pp. 301-312. Sesimbra, Portugal: Springer-Verlag.
78. Salzberg, S. L. (1991). A nearest hyperrectangle learning method. Machine Learning 6: 251-276.
79. Satoh, K. & Okamoto, S. (1994). Toward PAC-learning of weights from qualitative distance information. In D. W. Aha (ed.) Case-Based Reasoning: Papers from the 1994 Workshop (Technical Report WS-94-01). Menlo Park, CA: AAAI Press.
80. Schaffer, C. (1994). A conservation law for generalization performance. In Proceedings of the Eleventh International Conference on Machine Learning, pp. 259-265. New Brunswick, NJ: Morgan Kaufmann.
81. Schlimmer, J. C. (1987). Incremental adjustment of representations for learning. In Proceedings of the Fourth International Workshop on Machine Learning, pp. 79-90. Irvine, CA: Morgan Kaufmann.
82. Shannon, C. E. (1948). A mathematical theory of communication. Bell Systems Technology Journal 27: 379-423.
83. Short, R. & Fukunaga, K. (1980). A new nearest neighbor distance measure. In Proceedings of the Fifth International Conference on Pattern Recognition, pp. 81-86. Los Alamitos, CA: IEEE Press.
84. Short, R. & Fukunaga, K. (1981). The optimal distance measure for nearest neighbor classification. IEEE Transactions on Information Theory 27: 622-627.
85. Simard, P., Le Cun, Y. & Denker, J. (1993). Efficient pattern recognition using a new transformation distance. In Hanson, S. J., et al. (eds.), Advances in Neural Information Processing Systems 5. San Mateo, CA: Morgan Kaufmann.
86. Skalak, D. (1992). Representing cases as knowledge sources that apply local similarity metrics. In Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society, pp. 325-330. Bloomington, IN: Lawrence Erlbaum.
87. Skalak, D. (1994). Prototype and feature selection by sampling and random mutation hill climbing algorithms. In Proceedings of the Eleventh International Machine Learning Conference, pp. 293-301. New Brunswick, NJ: Morgan Kaufmann.
88. Stanfill, C. & Waltz, D. (1986). Toward memory-based reasoning. Communications of the Association for Computing Machinery 29: 1213-1228.
89. Tan, M. (1993). Cost-sensitive learning of classification knowledge and its application in robotics. Machine Learning 13: 7-34.
90. Ting, K. M. (1994). Discretization of continuous-valued attributes and instance-based learning (Technical Report). Sydney, Australia, University of Sydney, Basser Department of Computer Science.
91. Turney, P. D. (1993). Exploiting context when learning to classify. In Proceedings of the European Conference on Machine Learning, pp. 402-407. Vienna, Austria: Springer-Verlag.
92. Turney, P. D. (1995). Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research 2: 369-409.
93. Tversky, A. (1977). Features of similarity. Psychological Review 84: 327-352.
94. Utgoff, P. E. (1989). Incremental induction of decision trees. Machine Learning 4: 161-186.
95. van den Bosch, A. & Daelemans, W. (1993). Data-oriented methods for grapheme-to-phoneme conversion (Technical Report 42). Tilburg, Netherlands: Tilburg University, Institute for Language Technology and Artificial Intelligence.
96. Vapnik, V. (1992). Principles of risk minimization for learning theory. In R. P. Lippmann & J. E. Moody (eds.), Advances in Neural Information Processing Systems 3. Denver, CO: Morgan Kaufmann.
97. Volper, D. J. & Hampson, S. E. (1987). Learning and using specific instances. Biological Cybernetics 57: 57-71.
98. Weiss, S. M. & Kapouleas, I. (1989). An empirical comparison of pattern recognition, neural nets, and machine learning classification methods. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, pp. 781-787. Detroit, MI: Morgan Kaufmann.
99. Weiss, S. M. & Kulikowski, C. A. (1991). Computer systems that learn: Classification and prediction methods from statistics, neural nets, machine learning, and expert systems. San Mateo, CA: Morgan Kaufmann.
100. Wettschereck, D. (1994). A study of distance-based machine learning algorithms. Doctoral dissertation, Department of Computer Science, Oregon State University, Corvallis, OR. Available via WWW at http://nathan.gmd.de/persons/dietrich.wettschereck.html
101. Wettschereck, D. (1995a). A description of the mutual information approach and the variable similiarty metric (Technical Report 944). Sankt Augustin, Germany, German National Research Center for Computer Science, Artificial Intelligence Research Division.
102. Wettschereck, D. (1995b). Weighted kNN versus majority kNN: A recommendation (Technical Report 943). Sankt Augustin, Germany, German National Research Center for Computer Science, Artificial Intelligence Research Division.
103. Wettschereck, D. & Dietterich, T. G. (1992). Improving the performance of radial basis function networks by learning center locations. In J. Moody, S. Hanson, & R. Lippmann (eds.), Neural Information Processing Systems 4. Denver, CO: Morgan Kaufmann.
104. Wettschereck, D. & Dietterich, T. G. (1995). An experimental comparison of the nearest neighbor and nearest hyperrectangle algorithms. Machine Learning 19: 5-28.
105. Wolpert, D. H. (1990). Constructing a generalizer superior to NETtalk via a mathematical theory of generalization. Neural Networks 3: 445-452.
106. Wolpert, D. H. (1994). Personal communication.
107. Yau, H. C. & Manry, M. T. (1991). Iterative improvement of a nearest neighbor classifier. Neural Networks 4: 517-524.
108. Zhang, J. (1990). A method that combines inductive learning with exemplar-based learning. In Proceedings for Tools for Artificial Intelligence, pp. 31-37. Herndon, VA: IEEE Computer Society Press.