Tree induction vs. Logistic regression: A learning-curve analysis

Journal of Machine Learning Research

Volumn 4, Issue 2, 2004, Pages 211-255

  Perlich, Claudia ^a   Provost, Foster ^a   Simonoff, Jeffrey S ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Decision trees; Learning curves; Logistic regression; ROC analysis; Tree induction

Indexed keywords

DECISION TREES; LEARNING CURVES; LEARNING-CURVE ANALYSIS; LOGISTIC REGRESSION;

ALGORITHMS; DECISION THEORY; LOGISTICS; MATHEMATICAL MODELS; PROBABILITY; REGRESSION ANALYSIS; TREES (MATHEMATICS);

LEARNING SYSTEMS;

EID: 1242268938     PISSN: 15324435     EISSN: None     Source Type: Journal    
DOI: 10.1162/153244304322972694     Document Type: Article

References (68)

1. H. Akaike. A new look at statistical model identification. IEEE Transactions on Automatic Control, AU-19:716-722, 1974.
2. E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: bagging, boosting and variants. Machine Learning, 36:105-142, 1999.
3. C. Blake and C.J. Merz. UCI repository of machine learning databases. Technical Report, University of California, Irvine. Available electronically at http://www.ics.uci.edu/mlearn/MLRepository.html, 2000.
4. D.A. Bloch, R.A. Olshen, and M.G. Walker. Risk estimation for classification trees. Journal of Computational and Graphical Statistics 11(2):263-288, 2002.
5. G.E.P. Box. Robustness in the strategy of scientific model building. In Robustness in Statistics, eds. R.L. Launer and G.N. Wilkinson, pages 201-236, Academic Press, New York, 1979.
6. J. Bradford, C. Kunz, R. Kohavi, C. Brunk, and C.E. Bradley. Pruning decision trees with misclassification costs. In Proceedings of the Tenth European Conference on Machine Learning (ECML-98), pages 131-136, 1998.
7. A.P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30:1145-1159, 1997.
8. D. Brain and G. Webb. On the effect of data set size on bias and variance in classification learning. In Proceedings of the Fourth Australian Knowledge Acquisition Workshop, University of New South Wales, pages 117-128, 1999.
9. L. Breiman. Bagging predictors. Machine Learning, 24:123-140, 1996.
10. P. Bühlman and B. Yu. Analyzing Bagging. Annals of Statistics 30:927-961, 2002.
11. W. Buntine. A Theory of Learning Classification Rules. PhD thesis, School of Computer Science, University of Technology, Sydney, Australia, 1991.
12. K.P. Burnham and D.R. Anderson. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd. ed., Springer-Verlag, New York, 2002.
13. J. Catlett. Megainduction: Machine learning on Very Large Databases. PhD thesis, School of Computer Science, University of Technology, Sydney, Australia, 1991.
14. H.A. Chipman, E.I. George, and R.E. McCulloch. Bayesian CART model search (with discussion). Journal of the American Statistical Association, 93:935-948, 1998.
15. P. Clark and R. Boswell. Rule induction with CN2: some recent improvements. In Proceedings of the Sixth European Working Session on Learning, pages 151-163, Springer-Verlag, Porto, Portugal, 1991.
16. W.W. Cohen. Fast effective rule induction. Machine Learning: Proceedings of the Twelfth International Conference, eds. A. Prieditis and S. Russell, pages 115-123, Lake Tahoe, California, Morgan Kaufmann, 1995.
17. C. Cortes, L.D. Jackel, S.A. Solla, V. Vapnik, and J.S. Denker. Learning curves: asymptotic values and rate of convergence. Advances in Neural Information Processing Systems, Volume 6, pages 327-334, Morgan Kaufmann, San Mateo, California, 1994.
18. M.W. Craven and J.W. Shavlik. Learning to represent codons: a challenge problem for constructive induction. In Proceedings of the Thirteenth International Conference on Artificial Intelligence, Chambery, France, pages 1319-1324, 1993.
19. A. Danyluk and F. Provost. Telecommunications network diagnosis. In Handbook of Knowledge Discovery and Data Mining, eds. W. Kloesgen and J. Zytkow, Oxford University Press, Oxford, pp. 897-902, 2002.
20. P. Domingos and M. Pazzani. On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29:103-130, 1997.
21. B. Efron and R. Tibshirani. An Introduction to the Bootstrap, Chapman and Hall, New York, 1993.
22. M. Ennis, G. Hinton, D. Naylor, M. Revow, and R. Tibshirani. A comparison of statistical learning methods on the GUSTO database. Statist. Med. 17:2501-2508, 1998.
23. B.W. Flury and M.J. Schmid. Error rates in quadratic discrimination with constraints on the covariance matrices. Journal of Classification, 11:101-120, 1994.
24. C.J. Fombrun, N. Gardberg, and J. Sever. The reputation quotient: a multi-stakeholder measure of corporate reputation. Journal of B rand Management, 7:241-255, 2000.
25. J.H. Friedman. On bias, variance, 0/1-loss, and the curse of dimensionality. Data Mining and Knowledge Discovery, 1:55-77, 1997.
26. N. Friedman and M. Goldszmidt. Learning Bayesian networks with local structure. In Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, pages 252-262, Morgan Kaufmann, Portland, Oregon, 1996.
27. D.J. Hand. Construction and Assessment of Classification Rules, John Wiley and Sons, Chichester, 1997.
28. J.A. Hanley and B.J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143:29-36, 1982.
29. C. Harris-Jones and T.L. Haines. Sample size and misclassification: is more always better? AMSCAT-WP-97-118, AMS Center for Advanced Technologies, 1997.
30. T.J. Hastie and D. Pregibon. Shrinking Trees. AT&T Laboratories Technical Report, 1990.
31. D. Haussler, M. Kearns, H.S. Seung, and N. Tishby. Rigorous learning curve bounds from statistical mechanics. Machine Learning 25:195-236, 1996.
32. A.E. Hoerl and R.W. Kennard. Ridge regression: biased estimates for nonorthogonal problems. Technometrics, 12:55-67, 1970.
33. A.E. Hoerl, R.W. Kennard, and K.F. Baldwin. Ridge regression: some simulations. Communications in Statistics, 4:105-124, 1975.
34. D.W. Hosmer and S. Lemeshow. Applied Logistic Regression, 2nd. ed., John Wiley and Sons, New York, 2000.
35. R. Ihaka and R. Gentleman. R: a language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5:299-314, 1996.
36. T.S. Jaakkola and M.I. Jordan. Bayesian logistic regression: a variational approach. Statistics and Computing, 10:25-37, 2000.
37. D. Kibler and P. Langley. Machine learning as an experimental science. Proceedings of the Third European Working Session on Learning, pages 81-92, Pittman, Glasgow, 1988.
38. R. D. King, C. Feng, and A. Sutherland. STATLOG: comparison of classification algorithms on large real-world problems. Applied Artificial Intelligence, 9(3):289-334, 1995.
39. R. Kohavi. A study of cross-validation and boostrap for accuracy estimation and model selection. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1137-1143, 1995.
40. R. Kohavi. Scaling up the accuracy of naive-Bayes classifiers: a decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pages 202-207, 1996.
41. J.F. Lawless and K. Singhai. Efficient screening of nonnormal regression models. Biometrics, 34:318-327, 1978.
42. M. LeBlanc and R. Tibshirani. Monotone shrinkage of trees. Journal of Computational and Graphical Statistics, 7:417-433, 1998.
43. S. le Cessie and J.C. van Houwelingen. Ridge estimators in logistic regression. Applied Statistics, 41:191-201, 1992.
44. T.S. Lim, W. Y. Loh, and Y.S. Shih. A comparison of prediction accuracy, complexity, and training time for thirty-three old and new classification algorithms. Machine Learning, 40:203-228, 2000.
45. P. McCullagh and J.A. Nelder. Generalized Linear Models, 2nd. ed., Chapman and Hall, London, 1989.
46. T. Niblett. Constructing decision trees in noisy domains. In Proceedings of the Second European Working Session on Learning, pages 67-78, Sigma, Bled, Yugoslavia, 1987.
47. A. Ng and M. Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In Advances in Neural Information Processing Systems (NIPS-2001) 14: 841-848, 2001.
48. T. Oates and D. Jensen. The effects of training set size on decision tree complexity. In Machine Learning: Proceedings of the Fourteenth International Conference, ed. D. Fisher, pages 254-262, Morgan Kaufmann, San Mateo, California, 1997.
49. M. Pazzani, C. Merz, P. Murphy, K. Ali, T. Hume, and C. Brunk. Reducing misclassification costs. In Proceedings of the Eleventh International Conference on Machine Learning, pages 217-225, Morgan Kaufmann, San Mateo, California, 1994.
50. F. Provost and P. Domingos. Tree induction for probability-based rankings. Machine Learning, 52:3, forthcoming.
51. F. Provost and T. Fawcett. Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD-97), pages 43-48, AAAI Press, Meno Park, California, 1997.
52. F. Provost and T. Fawcett. Robust classification for imprecise environments. Machine Learning, 42:203-231, 2001.
53. F. Provost, T. Fawcett, and R. Kohavi. The case against accuracy estimation for comparing induction algorithms. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 445-453, Morgan Kaufmann, San Mateo, California, 1998.
54. F. Provost, D. Jensen, and T. Oates. Efficient progressive sampling. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 23-32, Association for Computing Machinery, New York, 1999.
55. F. Provost and V. Kolluri. A survey of methods for scaling up inductive algorithms. Data Mining and Knowledge Discovery, 3:131-169, 1999.
56. J.R. Quinlan. C4.5: Programs for Machine Learning, Morgan Kaufmann, San Mateo, California, 1993.
57. SAS Institute. SAS/STAT User's Guide, Version 8, SAS Publishing, Gary, North Carolina, 2000.
58. G. Schwarz. Estimating the dimension of a model. Annals of Statistics, 6:461-464, 1978.
59. J.W. Shavlik, R.J. Mooney, and G.G. Towell. Symbolic and neural learning algorithms: an experimental comparison. Machine Learning, 6:111-143, 1991.
60. Y. So. A tutorial on logistic regression. Technical Note 450. Available electronically at http://www.sas.com/service/techsup/tnote/tnote_index4.html, 1995.
61. J.R. Sobehart, R.M. Stein, V. Mikityanskaya, and L. Li. Moody's public firm risk model: a hybrid approach to modeling short term default risk. Technical Report, Moody's Investors Service, Global Credit Research. Available electronically at http://www.moodysqra.com/researoh/crm/53853.asp, 2000.
62. D. Steinberg and N.S. Cardell. The Hybrid CART-Logit Model in Classification and Data Mining. Salford Systems White Paper, 1998.
63. J. Swets. Measuring the accuracy of diagnostic systems. Science, 240:1285-1293, 1988.
64. W.N. Venables and B.D. Ripley. Modern Applied Statistics with S-Plus, 3rd. ed., Springer-Verlag, New York, 1999.
65. T.L.H. Watkin and A. Rau and M. Biehl. The statistical mechanics of learning a rule. Reviews of Modern Physics 65:499-556, 1993.
66. B.M. Wiesenfeld, J. Brockner, and C. Martin. A self-affirmation analysis of survivors' reactions to unfair organizational downsizings. Journal of Experimental Social Psychology, 35:441-460, 1999.
67. S.M. Weiss and N. Indurkhya. Rule-based machine learning methods for functional prediction. Journal of Artificial Intelligence Research, 3:383-403, 1995.
68. B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML-01), eds. C. Brodley and A. Danyluk, pages 609-616, Morgan Kaufmann, San Mateo, California, 2001.