Towards more practical average bounds on supervised learning

IEEE Transactions on Neural Networks

Volumn 7, Issue 4, 1996, Pages 953-968

  Gu, Hanzhong ^a   Takahashi, Haruhisa ^a  

^a UNIVERSITY OF ELECTRO COMMUNICATIONS   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION THEORY; COMPUTER SIMULATION; LEARNING ALGORITHMS; MATHEMATICAL MODELS; NEURAL NETWORKS; PERFORMANCE; REGRESSION ANALYSIS;

GENERALIZATION; HYPOTHESIS TESTING INEQUALITIES; LEARNING CURVES;

LEARNING SYSTEMS;

EID: 0030195241     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.508938     Document Type: Article

References (38)

1. V. N. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982.
2. V. N. Vapnik and A. Y. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory Probability Applicat., vol. 16, no. 2, pp. 264-280, 1971.
3. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, "Learnability and the Vapnik-Chervonenkis dimension," J. Assoc. Comput. Machinery, vol. 36, no. 4, pp. 929-965, 1989.
4. S. Amari, "A universal theorem on learning curves," Neural Networks, vol. 6, pp. 161-166, 1993.
5. S. Amari, N. Fujita, and S. Shinomoto, "Four types of learning curves," Neural Computa., vol. 4, no. 4, pp. 605-618, 1992.
6. D. Haussler, M. Kearns, M. Opper, and R. Schapire, "Estimating average-case learning curves using Bayesian, statistical physics, and VC dimension methods," Advances Neural Inform. Processing Syst., vol. 4, pp. 855-862, 1992.
7. D. Haussler, M. Kearns, H. S. Seung, and N. Tishby, "Rigorous learning curve bounds from statistical mechanics," in Proc. 7th Annu. ACM Wkshp. Comput. Learning Theory (COLT), 1994, pp. 76-87.
8. E. Levin, N. Tishby, and S. Solla, "A statistical approach to learning and generalization in layered neural networks," Proc. IEEE, vol. 78, no. 10, pp. 1568-1574, 1990.
9. H. Sompolinsky, H. S. Seung, and N. Tishby, "Learning curves in large neural networks," in Proc. 4th Annu. Wkshp. Comput. Learning Theory (COLT), 1991, pp. 112-127.
10. S. B. Holden and M. Niranjan, "On the practical applicability of VC dimension bounds," Neural Computa., vol. 7, no. 6, pp. 1265-1288, 1995.
11. S. Geman, E. Bienenstock, and R. Doursat, "Neural networks and the bias/variance dilemma," Neural Computa., vol. 4, no. 1, pp. 1-58, 1992.
12. H. Akaike, "Statistical predictor identification," Ann. Inst. Statist. Math., vol. 22, pp. 203-217, 1970.
13. A. Barron, "Predicted squared error: A criterion for automatic model selection," in Self-Organizing Methods in Modeling, S. Farlow, Ed. New York: Marcel Dekker, 1984.
14. R. L. Eubank, Spline Smoothing and Nonparametric Regression. New York: Marcel Dekker, 1988.
15. J. E. Moody, "The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems," Advances Neural Inform. Processing Syst., vol. 4, pp. 847-854, 1992.
16. D. Mackay, "Bayesian model comparison and backprop nets," Advances Neural Inform. Processing Syst., vol. 4, pp. 839-846, 1992.
17. D. Haussler, "Decision theoretic generalizations of the PAC model for neural nets and other learning applications," Inform. Computa., vol. 100, pp. 78-150, 1992.
18. M. Kearns, R. E. Schapire, and L. M. Sellie, "Toward efficient agnostic learning," in Proc. 5th ACM Wkshp. Computa. Learning Theory, 1992, pp. 341-352.
19. L. G. Valiant, "A theory of the learnable," Comm. ACM, vol. 27, no. 11, pp. 1134-1142, 1984.
20. M. Stone, "Cross-validation choice and assessment of statistical predictions (with discussion)," J. Royal Statist. Soc. B, vol. 36, pp. 111-147, 1974.
21. B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York: Chapman and Hall, 1993.
22. D. Pollard, Convergence of Stochastic Processes. New York: Springer-Verlag, 1984.
23. A. Macintyre and E. Sontag, "Finiteness results for sigmoidal 'neural' networks," in Proc. 25th Annu. Symp. Theory Comput., 1993, pp. 325-334.
24. L. Saitta and F. Bergadano, "Pattern recognition and Valiant's learning framework," IEEE Trans. Pattern Anal. Machine Intell., vol. 15, pp. 145-155, 1993.
25. M. Anthony and N. Biggs, Computational Learning Theory. Cambridge, U.K.: Cambridge Univ. Press, 1992.
26. H. Seung, H. Sompolinsky, and N. Tishby, "Learning from examples in large neural networks," Phys. Rev., vol. A45, pp. 6058-6091, 1992.
27. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Contr., Signals, Syst., vol. 2, no. 4, pp. 303-314, 1989.
28. H. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, pp. 359-366, 1989.
29. A. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, pp. 930-945, 1993.
30. D. W. Ruck, S. K. Rogers, M. Kabrisky, M. E. Oxley, and B. W. Suter, "The multilayer perceptron as an approximation to a Bayes optimal discriminant function," IEEE Trans. Neural Networks, vol. 1, pp. 296-298, 1990.
31. E. Wan, "Neural network classification: A Bayesian interpretation," IEEE Trans. Neural Networks, vol. 1, pp. 303-305, 1990.
32. P. Koiran and E. Sontag, "Neural networks with quadratic VC dimension," NeuroCOLT Tech. Rep. NC-TR-95-044, 1995.
33. D. Cohn and G. Tesauro, "How tight are the Vapnik-Chervonenkis bounds?" Neural Computa., vol. 4, pp. 249-269, 1992.
34. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," in Parallel Distributed Processing, vol. 1. Cambridge, MA: MIT Press, 1986, pp. 318-362.
35. A. Namatame and Y. Kimata, "Improving the generalizing capabilities of a backpropagation network," Int. J. Neural Networks Res. Applicat., vol. 1, no. 2, pp. 86-94, 1989.
36. M. P. Perrone and L. N. Cooper, "When networks disagree: Ensemble methods for hybrid neural networks," in Neural Networks Speech Image Processing, R. J. Mammone, Ed. New York: Chapman and Hall, 1993.
37. S. Hashem, "Optimal linear combinations of neural networks," Ph.D. dissertation, Purdue Univ., West Lafayette, IN, 1993.
38. R. Meir, "Bias, variance and the combination of estimators: The case of linear least squares," preprints, Dep. Electrical Eng., Technion, Haifa, Israel, 1994.