An Experimental and Theoretical Comparison of Model Selection Methods

Machine Learning

Volumn 27, Issue 1, 1997, Pages 7-50

  Kearns, Michael ^a   Mansour, Yishay ^b   Ng, Andrew Y ^c   Ron, Dana ^d  

^a AT AND T LABS RESEARCH   (United States)

^b TEL AVIV UNIVERSITY   (Israel)

^c Carnegie Mellon University ^*  (United States)

^d MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Complexity regularization; Cross validation; Minimum description length principle; Model selection; Structural risk minimization; vc dimension

Indexed keywords

BOOLEAN FUNCTIONS; COMPUTATIONAL COMPLEXITY; ERROR ANALYSIS; PROBLEM SOLVING; RANDOM PROCESSES; THEOREM PROVING;

CROSS VALIDATION (CV); GUARANTEED RISK MINIMIZATION (GRM); MINIMUM DESCRIPTION LENGTH PRINCIPLE (MDL); MODEL SELECTION METHODS;

LEARNING SYSTEMS;

EID: 0031122049     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1007344726582     Document Type: Review

References (19)

1. Barron, A. R., & Cover, T. M. (1991). Minimum complexity density estimation. IEEE Transactions on Information Theory, 37, 1034-1054.
2. Blum, A., & Rivest R. L. (1989). Training a 3-node neural net is NP-Complete. In David S. Touretzky, editor, Advances in Neural Information Processing Systems I, (pp. 494-501). Morgan Kaufmann, San Mateo, CA.
3. Cover T., & Thomas J. (1991). Elements of Information Theory. Wiley.
4. Haussler, D., & Kearns, M., & Seung, S., & Tishby, N. (1994). Rigourous learning curve bounds from statistical mechanics. In Proceedings of the Seventh Annual ACM Confernce on Computational Learning Theory, (pp. 76-87).
5. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58, 13-30.
6. Kearns, M. (1995). A bound on the error of cross validation, with consequences for the training-test split. In Advances in Neural Information Processing Systems 8. The MIT Press.
7. Kearns, M., & Schapire, R., & Sellie, L. (1992). Toward efficient agnostic learning. In Proceedings of the 5th Annual Workshop on Computational Learning Theory, (pp. 341-352).
8. Pitt, L., & Valiant, L. (1988). Computational limitations on learning from examples. Journal of the ACM, 35, 965-984.
9. Quinlan, J., & Rivest, R. (1989). Inferring decision trees using the minimum description length principle. Information and Computation, 80, 227-248.
10. Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14, 465-471.
11. Rissanen, J. (1986). Stochastic complexity and modeling. Annals of Statistics, 14, 1080-1100.
12. Rissanen, J. (1989). Stochastic Complexity in Statistical Inquiry, volume 15 of Series in Computer Science. World Scientific.
13. Schaffer, C. (1994). A conservation law for generalization performance. In Proceedings of the Eleventh International Conference on Machine Learning, (pp. 259-265).
14. Seung, H. S., & Sompolinsky, H., & Tishby, N. (1992). Statistical mechanics of learning from examples. Physical Review, A45, 6056-6091.
15. tone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society B, 36, 111-147.
16. tone, M. (1977). Asymptotics for and against cross-validation. Biometrika, 64, 29-35.
17. Vapnik, V. (1982). Estimation of Dependences Based on Empirical Data. Springer-Verlag.
18. Vapnik, V., & Chervonenkis, A. (1971). On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16, 264-280.
19. Wolpert, D. (1992). On the connection between in-sample testing and generalization error. Complex Systems, 6, 47-94.