Asymptotic statistical theory of overtraining and cross-validation

IEEE Transactions on Neural Networks

Volumn 8, Issue 5, 1997, Pages 985-996

  Amari, Shun Ichi ^a,b,c   Murata, Noboru ^c   Müller, Klaus Robert ^c,d   Finke, Michael ^e   Yang, Howard Hua ^a,b  

^a IEEE   (Japan)

^b RIKEN   (Japan)

^c UNIVERSITY OF TOKYO   (Japan)

^d FRAUNHOFER FIRST   (Germany)

^e UNIVERSITY OF KARLSRUHE   (Germany)

Author keywords

Asymptotic analysis; Cross validation; Early stopping; Generalization; Overtraining; Stochastic neural networks

Indexed keywords

COMPUTER SIMULATION; ERROR CORRECTION; RANDOM PROCESSES; STATISTICAL METHODS;

ASYMPTOTIC STATISTICAL THEORY; KULLBACK LEIBLER DIVERGENCE; STOCHASTIC NEURAL NETWORKS;

NEURAL NETWORKS;

EID: 0031236925     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.623200     Document Type: Article

References (34)

1. S. Amari, "Theory of adaptive pattern classifiers," IEEE Trans. Electron. Comput., vol. EC-16, pp. 299-307, 1967.
2. _, "A universal theorem on learning curves," Neural Networks, vol. 6, pp. 161-166, 1993.
3. S. Amari and N. Murata, "Statistical theory of learning curves under entropie loss criterion," Neural Computa., vol. 5, pp. 140-153, 1993.
4. S. Amari, N. Murata, K.-R. Müller, M. Finke, and H. Yang, "Statistical theory of overtraining - Is cross-validation effective?," in NIPS'95: Advances in Neural Information Processing Systems 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996.
5. H. Akaike, "A new look at statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, 1974.
6. D. Barber, D. Saad, and P. Sollich, "Test error fluctuations in finite linear perceptrons," Neural Computa. vol. 7, pp. 809-821, 1995.
7. _, "Finite-size effects and optimal test size in linear perceptrons," J. Phys. A, vol. 28, pp. 1325-1334, 1995.
8. N. Barkai, H. S. Seung, and H. Sompolinsky, "On-line learning of dichotomies," in Advances in Neural Information Processing Systems NIPS 7, G. Tesauro, D. S. Touretzky, and T. K. Leen, Eds. Cambridge, MA: MIT Press, 1995.
9. C. M. Bishop, "Regularization and complexity control in feedforward networks," Aston Univ., Tech. Rep. NCRG/95/022, 1995.
10. S. Bös, W. Kinzel, and M. Opper, "Generalization ability of perceptrons with continuous outputs," Phys. Rev., vol. E47, pp. 1384-1391, 1993.
11. _, "Avoiding overfitting by finite temperature learning and cross-validation," in Proc. Int. Conf. Artificial Neural Networks ICANN'95, Pans, 1995, pp. 111-116.
12. B. Efron and R. Tibshirani, An Introduction to the Bootstrap. London: Chapman and Hall, 1993.
13. M. Finke and K.-R. Müller, "Estimating a posteriori probabilities using stochastic network models," in Proc. 1993 Connectionist Models Summer School, M. Mozer, P. Smolensky, D. S. Touretzky, J. L. Elman, and A. S. Weigend, Eds. Hillsdale, NJ: Lawrence Erlbaum, 1994, p. 324.
14. I. Guyon 1996, "A scaling law for lhe validalion-set training-set ratio," preprint.
15. M. H. Hassoun, Fundamentals of Artificial Neural Networks. Cambodge, MA: MIT Press, 1995.
16. R. Hecht-Nielsen, Neurocomputing. Reading, MA: Addison-Wesley, 1989.
17. T. Heskes and B. Kappen, Learning process in neural networks, Phys. Rev., vol. A44, pp. 2718-2762, 1991.
18. M. Kearns, "A bound on the error of cross validation using the approximation and estimation rates, with consequences for the training-test split," in NIPS'95: Advances in Neural Information Processing Systems 8, D. S. Touretzky, M. C. Mozer and M. E. Hasselmo, Eds. Cambridge, MA: MIT Press, 1996.
19. A. Krogh and J. Hertz, "Generalization in a linear perceptron in the presence of noise," J. Phys. A, vol. 25, pp. 1135-1147, 1992.
20. J. E. Moody, "The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems," in NIPS 4: Advances in Neural Information Processing Systems, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. San Mateo, CA: Morgan Kaufmann, 1992.
21. N. Murata, S. Yoshizawa, and S. Amari, "A criterion for determining the number of parameters in an artificial neural-network model," in Artificial Neural Networks, T. Kohonen, et al., Eds.. Amsterdam, The Netherlands: Elsevier, 1991, pp. 9-14.
22. _, "Learning curves, model selection and complexity of neural networks," in NIPS 5: Advances in Neural Information Processing Systems, S. J. Hanson et al., Eds. San Mateo, CA: Morgan Kaufmann, 1993.
23. _, "Network information criterion - Determining the number of hidden units for an artificial neural-network model," IEEE Trans. Neural Networks, vol. 5, pp. 865-872, 1994.
24. K.-R. Müller, M Finke, N. Murata, K Schulten, and S. Amari, "A numerical study on learning curves in stochastic multilayer feedforward networks," Univ. Tokyo, Tech. Rep. METR 03-95, 1995,
25. also Neural Computa., vol. 8, pp. 1085-1106, 1996.
26. T. Poggio and F. Girosi, "Regularizalion algorithms for learning that are equivalent to multilayer networks," Science, vol. 247, pp. 978-982, 1990.
27. J. Rissanen, "Stochastic complexity and modeling," Ann. Statist., vol. 14, pp. 1080-1100, 1986.
28. D. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing: Explorations in the Micro structure of Cognition, vol. 1 - Foundations. Cambridge, MA: MIT Press, 1986.
29. D. Saad, S. A. Solla, "On-line learning in soft committee machines," Phys. Rev. E, vol. 52, pp. 4225-4243,
30. and "Exact solution for on-line learning in multilayer neural nelworks," Phys. Rev. Lett., vol. 74, pp. 4337-4340, 1995.
31. J. Sjöberg and L. Ljung, "Overtraining, regularizalion and searching for minimum with application to neural networks," Linköping Univ., Sweden, Tech. Rep. LiTH-ISY-R-1567, 1994.
32. C. Wang, S. S. Venkatesh, J. S. Judd, "Optimal stopping and effective machine complexity in learning," to appear, 1994 (revised and extended version of NIPS vol. 6, pp. 303-310, 1995).
33. V. Vapnik, Estimation of Dependences Based on Emprirical Data. New York: Springer-Verlag, 1992.
34. _, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995.