Multiple comparisons in induction algorithms

Machine Learning

Volumn 38, Issue 3, 2000, Pages 309-338

  Jensen, David D ^a   Cohen, Paul R ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION THEORY; FUNCTIONS; LEARNING ALGORITHMS; STATISTICAL METHODS;

INDUCTION ALGORITHMS; MULTIPLE COMPARISON PROCEDURES (MCP);

LEARNING SYSTEMS;

EID: 0033907286     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007631014630     Document Type: Article

References (48)

1. Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. (1989). Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM, 36, 929-965.
2. Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and Regression Trees. Belmont, CA: Wadsworth International.
3. Brodley, C. & Rissland, E. (1993). Measuring concept change. Training Issues in Incremental Learning: Papers from the 1993 Spring Symposium (pp. 99-108). Menlo Park, CA: AAAI Press.
4. Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. Cambridge, MA: MIT Press.
5. Dietterich, T. (1995). Overfitting and under-computing in machine learning. ACM Computing Surveys, 27, 326-327.
6. Edgington, E. (1995). Randomization Tests (3rd edition). New York, NY: Marcel Dekker.
7. Einhorn, H. (1972). Alchemy in the behavioral sciences. Public Opinion Quarterly, 36, 367-378.
8. Fayyad, U. & Irani, K. (1992). The attribute selection problem in decision tree generation. Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI-92) (pp. 104-110). Menlo Park, CA: AAAI Press.
9. Feelders, A. & Verkooijen, W. (1996). On the statistical comparison of inductive learning methods. In D. Fisher & H.-J. Lenz (Eds.), Learning from Data: Artificial and Intelligence V. New York, NY: Springer Verlag.
10. Fisher, D. & Schlimmer, J. (1988). Concept simplification and prediction accuracy. Proceedings of the Fifth International Conference on Machine Learning (pp. 22-28). San Mateo, CA: Morgan Kaufmann.
11. Gaines, B. (1989). An ounce of knowledge is worth a Ion of data: Quantitative studies of the trade-off between expertise and data based on statistically well-founded empirical induction. Proceedings of the Sixth International Workshop on Machine Learning (pp. 156-159). San Mateo, CA: Morgan Kaufmann.
12. Gascuel, O. & Caraux, G. (1992). Statistical significance in inductive learning. Proceedings of the Tenth European Conference on Artificial Intelligence (pp. 435-439). Chichester: Wiley.
13. Geman, S., Bienenstock, E., & Doursat, R. (1992). Neural networks and the bias/variance dilemma. Neural Computation, 4, 1-58.
14. Hand, D. & Taylor, C. (1987). Multivariate Analysis of Variance and Repeated Measures: A Practical Approach for Behavioural Scientists. London: Chapman and Hall.
15. Hawkins, D. & Kass, G. (1982). Automatic interation detection. In D. Hawkins (Ed.), Topics in Applied Multivariate Analysis. Cambridge: Cambridge University Press.
16. Iba, W., Wogulis, J., & Langley, P. (1988). Trading off simplicity and coverage in incremental concept learning. Proceedings of the Fifth International Conference on Machine Learning (pp. 73-79). San Mateo, CA: Morgan Kaufmann.
17. Jensen, D. (1991). Knowledge discovery through induction with randomization testing. Proceedings of the 1991 Knowledge Discovery in Databases Workshop (pp. 148-159). Menlo Park, CA: AAAI.
18. Jensen, D. (1992). Induction with Randomization Testing: Decision-Oriented Analysis of Large Data Sets. Doctoral dissertation. St. Louis, MO: Washington University.
19. Jensen, D. & Schmill, M. (1997). Adjusting for multiple comparisons in decision tree pruning. Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (pp. 195-198). Menlo Park, CA: AAAI Press.
20. Kass, G. (1975). Significance testing in Automatic Interaction Detection (A.I.D.). Applied Statistics, 24, 178-189.
21. Kass, G. (1980). An exploratory technique for investigating large quantities of categorical data. Applied Statistics, 29, 119-127.
22. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. IJCAI: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (pp. 1137-1143). San Francisco, CA: Morgan Kaufmann.
23. Kohavi, R. & Wolpert, D. (1996). Bias plus variance decomposition for zero-one loss functions. Proceedings of the Thirteenth International Conference on Machine Learning (pp. 275-283). San Francisco, CA: Morgan Kaufmann.
24. Liu, W. & White, A. (1994). The importance of attribute selection measures in decision tree induction. Machine Learning, 15, 25-41.
25. Miller, R. (1981). Simultaneous Statistical Inference (2nd edition). New York, NY: Springer-Verlag.
26. Mingers, J. (1989a). An empirical comparison of pruning methods for decision tree induction. Machine Learning, 4, 227-243.
27. Mingers, J. (1989b). An empirical comparison of selection measures for decision-tree induction. Machine Learning, 3, 319-342.
28. Morgan, J. & Andrews, F. (1973). A comment on Einhorn's "Alchemy in the behavioral sciences". Public Opinion Quarterly, 37, 127-129.
29. Murthy, S. & Salzberg, S. (1995). Lookahead and pathology in decision tree induction. IJCAI: Proceedings of Fourteenth International Joint Conference on Artificial Intelligence (pp. 1025-1031). San Francisco, CA: Morgan Kaufmann.
30. Noreen, E. (1989). Computer-Intensive Methods for Testing Hypotheses: An Introduction. New York, NY: Wiley.
31. Oates, T. & Jensen, D. (1997). The effects of training set size on decision tree complexity. Proceedings of the Fourteenth International Conference on Machine Learning (pp. 254-262). San Francisco, CA: Morgan Kaufmann.
32. Pearl, J. (1978). On the connection between the complexity and credibility of inferred models. International Journal of General Systems, 4, 255-264.
33. Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81-106.
34. Quinlan, J. R. (1987). Simplifying decision trees. International Journal of Man-Machine Studies, 27, 221-234.
35. Quinlan, J. R. (1988). Decision trees and multi-valued attributes. In J. Hayes, D. Michie & J. Richards (Eds.), Machine Intelligence (Vol. 11). Oxford, England: Clarendon Press.
36. Quinlan, J. R. (1996). Improved use of continuous attributes in C4.5. Journal of Artificial Intelligence Research, 4, 77-90.
37. Quinlan, J. R. & Cameron-Jones, R. (1995). Oversearching and layered search in empirical learning. IJCAI: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (pp. 1019-1024). San Francisco, CA: Morgan Kaufmann.
38. Quinlan, J. R. & Rivest, R. (1989). Inferring decision trees using the minimum description length principle. Information and Computation, 80, 227-248.
39. Rao, R., Gordon, D., & Spears, W. (1995). For every generalization action, is there really an equal and opposite reaction? Analysis of the conservation law for generalization performance. Machine Learning: Proceedings of the Twelfth International Conference (pp. 471-479). San Francisco, CA: Morgan Kaufmann.
40. Ross, S. (1984). A First Course in Probability (2nd edition). New York, NY: Macmillan.
41. Salzberg, S. (1997). On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery, 1, 317-328.
42. Schaffer, C. (1993). Overfitting avoidance as bias. Machine Learning, 10, 153-178.
43. Schaffer, C. (1994). A conservation law for generalization performance. Proceedings of the Eleventh International Conference on Machine Learning (pp. 259-265). San Francisco, CA: Morgan Kaufmann.
44. Sonquist, J., Baker, E., & Morgan, J. (1971). Searching for Structure (Alias, AID-III); An Approach to Analysis of Substantial Bodies of Micro-Data and Documentation for a Computer Program (Successor to the Automatic Interaction Detector Program). Ann Arbor, MI: Survey Research Center, Institute for Social Research, The University of Michigan.
45. Weiss, S. & Kulikowski, C. (1991). Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. San Mateo, CA: Morgan Kaufmann.
46. White, A. & Liu, W. (1995). Superstitious learning and induction. Artificial Intelligence Review, 9, 3-18.
47. Wolpert, D. (1992). On the connection between in-sample testing and generalization error. Complex Systems, 6, 47-94.
48. Wolpert, D. (1994). Off-training set error and a priori distinctions between learning algorithms. Technical Report SFI TR 95-01-003. Santa Fe, NM: Santa Fe Institute.