Inference for the generalization error

Machine Learning

Volumn 52, Issue 3, 2003, Pages 239-281

  Nadeau, Claude ^a   Bengio, Yoshua ^b  

^a HEALTH CANADA   (Canada)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

Author keywords

Cross validation; Generalization error; Hypothesis tests; Power; Size; Variance estimation

Indexed keywords

ERROR ANALYSIS; LEARNING ALGORITHMS; LEARNING SYSTEMS; RANDOM PROCESSES; SET THEORY; STATISTICAL METHODS; THEOREM PROVING;

CROSS VALIDATION ESTIMATOR; GENERALIZATION ERROR; HYPOTHESIS TESTS;

INFERENCE ENGINES;

EID: 0042847140     PISSN: 08856125     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1024068626366     Document Type: Article

References (18)

1. Blake, C., Keogh, E., & Merz, C.-J. (1998). UCI repository of machine learning databases.
2. Breiman, L. (1996). Heuristics of instability and stabilization in model selection. Annals of Statistics, 24:6, 2350-2383.
3. Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and regression trees. Wadsworth International Group.
4. Burges, C. (1998). A tutorial and support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2:2, 1-47.
5. Devroye, L., Gyröfi, L., & Lugosi, G. (1996). A probabilistic theory of pattern recognition. Springer-Verlag.
6. Dietterich, T. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10:7, 1895-1924.
7. Efron, B., & Tibshirani, R. J. (1993). An introduction to the boostrap. Monographs and Statistics and Applied Probability 57, New York, NY: Chapman & Hall.
8. Everitt, B. (1977). The analysis of contingency tables. London: Chapman & Hall.
9. Goutte, C. (1997). Note on free lunches and cross-validation. Neural Computation, 9:6, 1053-1059.
10. Hinton, G., Neal, R., Tibshirani, R., & DELVE team members. (1995). Assessing learning procedures using DELVE. Technical report, University of Toronto, Department of Computer Science.
11. Kearns, M., & Ron, D. (1997). Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Tenth Annual Conference on Computational Learning Theory (pp. 152-162). Morgan Kaufmann.
12. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceeding of the Fourteenth International Joint Conference on Artificial Intelligence (pp. 1137-1143). Morgan Kaufmann.
13. Kolen, J. & Pollack, J. (1991). Back propagation in sensitive to initial conditions. Advances in Neural Information Processing Systems (pp. 860-867). San Francisco, CA: Morgan Kauffmann.
14. Nadeau, C., & Bengio, Y. (1999). Inference for the generalisation error. Technical report 99s-25, CIRANO.
15. Vapnik, V. (1982). Estimation of dependences based on empirical data. Berlin: Springer-Verlag.
16. White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1-25.
17. Wolpert, D., & Macready, W. (1995). No free lunch theorems for search. Technical report SFI-TR-95-02-010, The Santa Fe Institute.
18. Zhu, H., & Rohwer, R. (1996). No free lunch for cross validation. Neural Computation, 8:7, 1421-1426.