Decision trees do not generalize to new variations

Computational Intelligence

Volumn 26, Issue 4, 2010, Pages 449-467

  Bengio, Yoshua ^a   Delalleau, Olivier ^a   Simard, Clarence ^a  

^a UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Curse of dimensionality; Decision trees; Parity function

Indexed keywords

CURSE OF DIMENSIONALITY; DECISION TREE LEARNING ALGORITHM; DISTRIBUTED REPRESENTATION; INPUT SPACE; NON-PARAMETRIC STATISTICAL METHODS; PARITY FUNCTIONS; TRAINING DATA; TRAINING SETS;

ARTIFICIAL INTELLIGENCE; DECISION TREES;

LEARNING ALGORITHMS;

EID: 78649265006     PISSN: 08247935     EISSN: 14678640     Source Type: Journal    
DOI: 10.1111/j.1467-8640.2010.00366.x     Document Type: Article

References (32)

1. Ajtai, M., 1983. -formulae on finite structures. Annals of Pure and Applied Logic, 24 (1): 1 - 48.
2. Bengio, Y., 2009. Learning deep architectures for AI. (1995). Foundations and Trends in Machine Learning, 2 (1): 1 - 127.
3. Bengio, Y., and, Y. LeCun,. 2007. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. Edited by, L. Bottou, O. Chapelle, D. DeCoste, and, J. Weston,. MIT Press, Cambridge, MA.
4. Bengio, Y., O. Delalleau, and, N. Le Roux,. 2006a. The curse of highly variable functions for local kernel machines. In Advances in Neural Information Processing Systems 18 (NIPS05). Edited by, Y. Weiss, B. Schölkopf, and, J. Platt,. MIT Press, Cambridge, MA, pp. 107 - 114.
5. Bengio, Y., M. Monperrus, and, H. Larochelle,. 2006b. Non-local estimation of manifold structure. Neural Computation, 18 (10): 2509 - 2528.
6. Bengio, Y., H. Larochelle, and, P. Vincent,. 2006c. Non-local manifold parzen windows. In Advances in Neural Information Processing Systems 18 (NIPS05). Edited by, Y. Weiss, B. Schölkopf, and, J. Platt,. MIT Press, Cambridge, MA, pp. 115 - 122.
7. Bengio, Y., P. Lamblin, D. Popovici, and, H. Larochelle,. 2007. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems 19 (NIPS06). Edited by, B. Schölkopf, J. Platt, and, T. Hoffman,. MIT Press, Cambridge, MA, pp. 153 - 160.
8. Boser, B. E., I. M. Guyon, and, V. N. Vapnik,. 1992. A training algorithm for optimal margin classifiers. In Fifth Annual Workshop on Computational Learning Theory. ACM, Pittsburgh, PA, pp. 144 - 152.
9. Breiman, L., 2001. Random forests. Machine Learning, 45 (1): 5 - 32.
10. Breiman, L., J. H. Friedman, R. A. Olshen, and, C. J. Stone,. 1984. Classification and Regression Trees. Wadsworth International Group, Belmont, CA.
11. Cortes, C., and, V. Vapnik,. 1995. Support vector networks. Machine Learning, 20: 273 - 297.
12. Cucker, F., and, D. Grigoriev,. 1999. Complexity lower bounds for approximation algebraic computation trees. Journal of Complexity, 15 (4): 499 - 512.
13. Freund, Y., and, R. E. Schapire,. 1996. Experiments with a new boosting algorithm. In Machine Learning: Proceedings of Thirteenth International Conference. ACM, USA, pp. 148 - 156.
14. Grigoriev, D., M. Karpinski, and, A. C.-C. Yao,. 1995. An exponential lower bound on the size of algebraic decision trees for MAX. Electronic Colloquium on Computational Complexity (ECCC), 2 (057): 193 - 203.
15. Håstad, J., 1986. Almost optimal lower bounds for small depth circuits. In Proceedings of the 18th Annual ACM Symposium on Theory of Computing. ACM Press, Berkeley, California, pp. 6 - 20.
16. Håstad, J., and, M. Goldmann,. 1991. On the power of small-depth threshold circuits. Computational Complexity, 1: 113 - 129.
17. Hinton, G. E., 1986. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society. Lawrence Erlbaum, Hillsdale, NJ, Amherst, MA, pp. 1 - 12.
18. Hinton, G. E., and, Z. Ghahramani,. 1997. Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society of London, B (352): 1177 - 1190.
19. Hinton, G. E., S. Osindero, and, Y. Teh,. 2006. A fast learning algorithm for deep belief nets. Neural Computation, 18: 1527 - 1554.
20. Ho, T. K., 1995. Random decision forest. In 3rd International Conference on Document Analysis and Recognition (ICDAR95), Montreal, Canada, pp. 278 - 282.
21. Hornik, K., M. Stinchcombe, and, H. White,. 1989. Multilayer feedforward networks are universal approximators. Neural Networks, 2: 359 - 366.
22. Hutter, M., 2005. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer, Berlin.
23. Kolmogorov, A. N., 1965. Three approaches to the quantitative definition of information. Problems of Information and Transmission, 1 (1): 1 - 7.
24. Kong, E. B., and, T. G. Dietterich,. 1995. Error-correcting output coding corrects bias and variance. In International Conference on Machine Learning, pp. 313 - 321.
25. LeCun, Y., F.-J. Huang, and, L. Bottou,. 2004. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR04), Vol. 2. IEEE Computer Society, Los Alamitos, CA, pp. 97 - 104.
26. Li, M., and, P. Vitanyi,. 1997. An Introduction to Kolmogorov Complexity and Its Applications (2nd ed.). Springer, New York.
27. Loh, W.-Y., and, Y.-S. Shih,. 1997. Split selection methods for classification trees. Statistica Sinica, 7: 815 - 840.
28. Paccanaro, A., and, G. E. Hinton,. 2000. Extracting distributed representations of concepts and relations from positive and negative propositions. In International Joint Conference on Neural Networks (IJCNN), Como, Italy. IEEE, New York.
29. Pérez, E., and, L. A. Rendell,. 1996. Learning despite concept variation by finding structure in attribute-based data. In Proceedings of the Thirteenth International Conference on Machine Learning (ICML96). Edited by, L. Saitta,. Morgan Kaufmann, San Francisco, pp. 391 - 399.
30. Ranzato, M., C. Poultney, S. Chopra, and, LeCun, Y,. 2007. Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems 19 (NIPS06). Edited by, B. Schölkopf, J. Platt, and, T. Hoffman,. MIT Press, Cambridge, MA, pp. 1137 - 1144.
31. Solomonoff, R. J., 1964. A formal theory of inductive inference. Information and Control, 7: 1-22, 224-254.
32. Vilalta, R., G. Blix, and, L. Rendell,. 1997. Global data analysis and the fragmentation problem in decision tree induction. In Proceedings of the Ninth European Conference on Machine Learning (ECML97). Springer-Verlag, Berlin, pp. 312 - 327.