Minimax-optimal classification with dyadic decision trees

IEEE Transactions on Information Theory

Volumn 52, Issue 4, 2006, Pages 1335-1353

  Scott, Clayton ^a   Nowak, Robert D ^b  

^a Rice University   (United States)

^b University of Wisconsin Madison   (United States)

Author keywords

Complexity regularization; Decision trees; Feature rejection; Generalization error bounds; Manifold learning; Minimax optimality; Pruning; Rates of convergence; Recursive dyadic partitions; Statistical learning theory

Indexed keywords

COMPUTATIONAL COMPLEXITY; DECISION THEORY; OPTIMIZATION; STATISTICAL METHODS; TREES (MATHEMATICS);

COMPLEXITY REGULARIZATION; DYADIC DECISION TREES; FEATURE REJECTION; GENERALIZATION ERROR BOUNDS; MANIFOLD LEARNING; MINIMAX OPTIMAL CLASSIFICATION; MINIMAX OPTIMALITY; RATES OF CONVERGENCE; RECURSIVE DYADIC PARTITIONS; STATISTICAL LEARNING THEORY;

INFORMATION THEORY;

EID: 33645724205     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2006.871056     Document Type: Article

References (58)

1. A. B. Tsybakov, "Optimal aggregation of classifiers in statistical learning," Ann. Statist., vol. 32, no. 1, pp. 135-166, 2004.
2. V. Vapnik, Estimation of Dependencies Based on Empirical Data. New York: Springer-Verlag, 1982.
3. A. Barron, "Complexity regularization with application to artificial neural networks," in Nonparametric Functional Estimation and Related Topics. ser. NATO ASI, G. Roussas, Ed. Dordrecht, The Netherlands: Kluwer, 1991, pp. 561-576.
4. G. Lugosi and K. Zeger, "Concept learning using complexity regularization," IEEE Trans. Inf. Theory, vol. 42, no. 1, pp. 48-54, Jan, 1996.
5. C. Scott and R. Nowak, "Dyadic classification trees via structural risk minimization," in Advances in Neural Information Processing Systems, S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, vol. 15.
6. R. A. De Vore, "Nonlinear approximation," Acta Numerica, vol. 7, pp. 51-150, 1998.
7. A. Cohen, W. Dahmen, I. Daubechies, and R. A. DeVore, "Tree approximation and optimal encoding," Appl. Comput. Harmonic Anal., vol. 11, no. 2, pp. 192-226, 2001.
8. D. Donoho, "Wedgelets: Nearly minimax estimation of edges," Ann. Statist., vol. 27, pp. 859-897, 1999.
9. S. Mallat, A Wavelet Tour of Signal Processing. San Diego, CA: Academic, 1998.
10. E. Mammen and A. B. Tsybakov, "Smooth discrimination analysis," Ann. Statist., vol. 27, pp. 1808-1829, 1999.
11. J. S. Marron, "Optimal rates of convergence to Bayes risk in nonparametric discrimination," Ann. Statist., vol. 11, no. 4, pp. 1142-1155, 1983.
12. Y. Yang, "Minimax nonparametric classification - Part I: Rates of convergence," IEEE Trans. Inf. Theory, vol. 45, no. 7, pp. 2271-2284, Nov. 1999.
13. J.-Y. Audibert, "PAC-Bayesian statistical learning theory," Ph.D. dissertation, Université Paris 6, Paris, France, Jun. 2004.
14. A. B. Tsybakov and S. A. van de Geer, "Square root penalty: Adaptation to the margin in classification and in edge estimation," Ann. Statist., vol. 33, no. 3, pp. 1203-1224, 2005.
15. P. Bartlett, M. Jordan, and J. McAuliffe, "Convexity, classification, and risk bounds," J. Amer. Statist. Assoc.. Dep. Statistics, Univ. California, Berkeley, Tech. Rep. 638, 2003, to be published.
16. G. Blanchard, G. Lugosi, and N. Vayatis, "On the rate of convergence of regularized boosting classifiers," J. Mach. Learning Res., vol. 4, pp. 861-894, 2003.
17. J. C. Scovel and I. Steinwart, "Fast rates for support vector machines," Los Alamos Nat. Lab., Tech. Rep. LA-UR 03-9117, 2003.
18. Q. Wu, Y. Ying, and D. X. Zhou, "Multi-kernel regularized classifiers," J. Complexity, submitted for publication.
19. D. X. Zhou and K. Jetter, "Approximation with polynomial kernels and SVM classifiers," Adv. Comput. Math., 2004, submitted for publication.
20. C. Scott and R. Nowak, "Near-minimax optimal classification with dyadic classification trees," in Advances in Neural Information Processing Systems, S. Thrun, L. Saul, and B. Schölkopf, Eds. Cambridge, MA: MIT Press, 2004, vol. 16.
21. S. Murthy, "Automatic construction of decision trees from data: A multi-disciplinary survey," Data Mining and Knowledge Discovery, vol. 2, no. 4, pp. 345-389, 1998.
22. L. Breiman, J. Friedman, R. Olshen, and C. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth, 1984.
23. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann, 1993.
24. M. Kearns and Y. Mansour, "On the boosting ability of top-down decision tree learning algorithms," J. Comput. Syst. Sci., vol. 58, no. 1, pp. 109-128, 1999.
25. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996.
26. L. Gordon and R. Olshen, "Asymptotically efficient solutions to the classification problem," Ann. Statist., vol. 6, no. 3, pp. 515-533, 1978.
27. F. Esposito, D. Malerba, and G. Semeraro, "A comparitive analysis of methods for pruning decision trees," IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 5, pp. 476-491, May 1997.
28. Y. Mansour, "Pessimistic decision tree pruning," in Proc. 14th Int. Conf. Machine Learning, D. H. Fisher, Ed. Nashville, TN: Morgan Kaufmann, 1997, pp. 195-201.
29. M. Kearns and Y. Mansour, "A fast, bottom-up decision tree pruning algorithm with near-optimal generalization," in Proc. 15th Int. Conf. Machine Learning, J. W. Shavlik, Ed.. Madison, WI, 1998, pp. 269-277.
30. G. Blanchard, C. Schafer, and Y. Rozenholc, "Oracle bounds and exact algorithm for dyadic classification trees," in Learning Theory: 17th Annu. Conf. Learning Theory, COLT 2004, J. Shawe-Taylor and Y. Singer, Eds. Heidelberg, Germany: Springer-Verlag, 2004, pp. 378-392.
31. Y. Mansour and D. McAllester, "Generalization bounds for decision trees," in Proc. 13th Annu. Conf. Computational Learning Theory, N. Cesa-Bianchi and S. Goldman, Eds., Palo Alto, CA, 2000, pp. 69-74.
32. A. Nobel, "Analysis of a complexity based pruning scheme for classification trees," IEEE Trans. Inf. Theory, vol. 48, no. 8, pp. 2362-2368, Aug. 2002.
33. N. Berkman and T. Sandholm, "What Should be Minimized in a Decision Tree: A Re-Examination," Univ. Massachusetts at Amherst, Tech. Rep. TR 95-20, 1995.
34. M. Golea, P. Bartlett, W. S. Lee, and L. Mason, "Generalization in decision trees and DNF: Does size matter?," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 1998, vol. 10.
35. G. Blanchard, C. Schäfer, Y. Rozenholc, and K.-R. Müller, "Optimal dyadic decision trees," Tech. Rep., 2005. Available [Online] at http://www.math.u-psud.fr/~blanchard/; preprint.
36. D. Donoho, "CART and best-ortho-basis selection: A connection," Ann. Statist., vol. 25, pp. 1870-1911, 1997.
37. D. Donoho and I. Johnstone, "Ideal adaptation via wavelet shrinkage," Biometrika, vol. 81, pp. 425-455, 1994.
38. D. Donoho and I. Johnstone, "Adapting to unknown smoothness via wavelet shrinkage," J. Amer. Statist. Assoc., vol. 90, pp. 1200-1224, 1995.
39. D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard, "Wavelet shrinkage: Asymptopia?," J. Roy. Statist. Soc. B, vol. 57, no. 432, pp. 301-369, 1995.
40. E. Kolaczyk and R. Nowak, "Multiscale likelihood analysis and complexity penalized estimation," Ann. Statist., vol. 32, no. 2, pp. 500-527, 2004.
41. E. Kolaczyk and R. Nowak, "Multiscale generalized linear models for nonparametric function estimation," Biometrika, vol. 92, no. 1, pp. 119-133, 2005.
42. I. Johnstone, "Wavees and the theory of nonparametric function estimation," Phil. Trans. Roy. Soc. Lond. A., vol. 357, pp. 2475-2494, 1999.
43. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley, 1991.
44. S. Gey and E. Nedelec, "Risk bounds for CART regression trees," in MSRI Proc. Nonlinear Estimation and Classification. Berlin, Germany: Springer-Verlag, 2002.
45. A. W. van er Vaart and J. A. Wellner, Weak Convergence and Empirical Processes. New York: Springer-Verlag, 1996.
46. P. Bartlett, S. Boucheron, and G. Lugosi, "Model selection and error estimation," Machine Learning, vol. 48, pp. 85-113, 2002.
47. S. Boucheron, O. Bousquet, and G. Lugosi, Theory of Classification: A Survey of Recent Advances, 2004. Available [Online] at http:// www.econ.upf.es/~lugosi/; preprint.
48. R. Dudley, "Metric entropy of some classes of sets with differentiable boundaries," J. Approx. Theory, vol. 10, pp. 227-236, 1974.
49. K. Falconer, Fractal Geometry: Mathematical Foundations and Applications. West Sussex, U.K.: Wiley, 1990.
50. G. Blanchard, private communication, Aug. 2004.
51. K. Bennett, N. Cristianini, J. Shawe-Taylor, and D. Wu, "Enlarging the margins in perceptron decision trees," Machine Learning, vol. 41, pp. 295-313, 2000.
52. A. P. Korostelev and A. B. Tsybakov, Minimax Theory of Image Reconstruction. New York: Springer-Verlag, 1993.
53. E. Candes and D. Donoho, "Curvelets and curvilinear integrals," J. Approx. Theory, vol. 113, pp. 59-90, 2000.
54. T. Hagerup and C. Rüb, "A guided tour of chernoff bounds," Inf. Processing Lett., vol. 33, no. 6, pp. 305-308, 1990.
55. H. Chernoff, "A measure of asymptotic efficiency of tests of a hypothesis based on the sum of observations," Ann. Math. Statist., vol. 23, pp. 493-507, 1952.
56. M. Okamoto, "Some inequalities relating to the partial sum of binomial probabilites," Anna. Inst. Statist. Math., vol. 10, pp. 29-35, 1958.
57. B. Yu, "Assouad, Fano, and Le Cam," in Festschrift for Lucien Le Cam, D. Pollard, E. Torgersen, and G. Yang, Eds. Berlin, Germany: Springer-Verlag, 1997, pp. 423-435.
58. C. Huber, "Lower bounds for function estimation," in Festschrift for Lucien Le Cam, D. Pollard, E. Torgersen, and G. Yang, Eds. Berlin, Germany: Springer-Verlag, 1997, pp. 245-258.