The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network

IEEE Transactions on Information Theory

Volumn 44, Issue 2, 1998, Pages 525-536

  Bartlett, Peter L ^a,b  

^a IEEE

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Computational learning theory; Neural networks; Pattern recognition; Scale sensitive dimensions; Weight decay

Indexed keywords

BACKPROPAGATION; COMPUTATIONAL COMPLEXITY; ERROR ANALYSIS; LEARNING ALGORITHMS; LEARNING SYSTEMS; PATTERN RECOGNITION; PROBABILITY; TRANSFER FUNCTIONS;

SCALE SENSITIVE DIMENSIONS; WEIGHT DECAY;

NEURAL NETWORKS;

EID: 0032028728     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.661502     Document Type: Article

References (41)

1. N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler, "Scale-sensitive dimensions, uniform convergence, and learnability," J. Assoc. Comput. Mach., 1997, to be published.
2. M. Anthony and P. Bartlett, "Function learning from interpolation," In Computational Learning Theory: EUROCOLT'95, 1995.
3. M. Anthony and J. Shawe-Taylor, "A result of Vapnik with applications," Discr. Appl. Math., vol. 47, pp. 207-217, 1993.
4. A. R. Barron, "Universal approximation bounds for superposition of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, pp. 930-945, 1993.
5. P. L. Bartlett, "For valid generalization, the size of the weights is more important than the size of the network," Neural Inform. Process. Syst., vol. 9, pp. 134-140, 1997.
6. P. L. Bartlett, S. R. Kulkarni, and S. E. Posner, "Covering numbers for real-valued function classes," IEEE Trans. Inform. Theory, vol. 43, pp. 1721-1724, Sept. 1997.
7. P. L. Bartlett and P. M. Long, "Prediction, learning, uniform convergence, and scale-sensitive dimensions," J. Comput. and Syst. Sci., 1998, to be published.
8. E. Baum and D. Haussler, "What size net gives valid generalization?" Neural Computation, vol. 1, no. 1, pp. 151-160, 1989.
9. S. Ben-David, N. Cesa-Bianchi, D. Haussler, and P. M. Long, "Characterizations of learnability for classes of {0, ⋯, n}-valued functions," J. Comput. Syst. Sci., vol. 50, no. 1, pp. 74-86, 1995.
10. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, "Learn-ability and the Vapnik-Chervonenkis dimension," J. Assoc. Comput. Mach., vol. 36, no. 4, pp. 929-965, 1989.
11. B. Boser, I. Guyon, and V. Vapnik, "A training algorithm for optimal margin classifiers," in Proc. 5th Annu. ACM Workshop on Computational Learning Theory, 1992, pp. 144-152.
12. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996.
13. A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant, "A general lower bound on the number of examples needed for learning," Inform. Computat., vol. 82, pp. 247-261, 1989.
14. Y. Freund and R. E. Schapire, "A decision-theoretic generalization of online learning and an application to boosting," in Computational Learning Theory: Second Europ. Conf., EUROCOLT'95, 1995, pp. 23-37.
15. _, "Experiments with a new boosting algorithm," in Machine Learning: Proc. 13th Int. Conf., 1996.
16. P. W. Goldberg and M. R. Jerrum, "Bounding the Vapnik-Chervonenkis dimension of concept classes parametrized by real numbers," Mach. Learning, vol. 18, no. 2/3, pp. 131-148, 1995.
17. L. Gurvits and P. Koiran, "Approximation and learning of convex superpositions," in Computational Learning Theory: EUROCOLT'95, 1995.
18. I. Guyon, B. Boser, and V. Vapnik, "Automatic capacity tuning of very large VC-dimension classifiers," in NIPS 5. Los Altos, CA: Morgan Kaufmann, 1993, pp. 147-155.
19. D. Haussler, "Decision theoretic generalizations of the PAC model for neural net and other learning applications," Inform. Computat., vol. 100, no. 1, pp. 78-150, 1992.
20. D. Haussler and P. M. Long, "A generalization of Sauer's lemma," J. Comb. Theory, Ser. A, vol. 71, no. 2, pp. 219-240, 1995.
21. J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation. Reading, MA: Addison-Wesley, 1991.
22. W. Hoeffding, "Probability inequalities for sums of bounded random variables," Amer. Statist. Assoc. J., vol. 58, pp. 13-30, 1963.
23. M. Horvath and G. Lugosi, "A data-dependent skeleton estimate and a scale-sensitive dimension for classification," Tech. Rep., Pompeu Fabra Univ., Dec. 1996.
24. L. K. Jones, "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training," Ann. Statist., vol. 20, no. 1, pp. 608-613, 1992.
25. M. Karpinski and A. Macintyre, "Quadratic VC-dimension bounds for sigmoidal networks," in Proc. 27th Annu. Symp. on the Theory of Computing, 1995.
26. M. J. Kearns and R. E. Schapire, "Efficient distribution-free learning of probabilistic concepts," in Proc. 31st Symp. on Foundations of Computer Science. Los Alamitos, CA: IEEE Computer Soc. Press, 1990, pp. 382-391.
27. A. N. Kolmogorov and V. M. Tihomirov, ε-entropy and ε-capacity of sets in function spaces," Amer. Math. Soc. Translations (2), vol. 17, pp. 277-364, 1961.
28. K. J. Lang and M. Witbrock, "Learning to tell two spirals apart," in Proc. 1988 Connectionist Models Summer School. Los Altos, CA: Morgan Kaufmann, 1988.
29. S. Lawrence, C. L. Giles, and A. C. Tsoi, "What size neural network gives optimal generalization? Convergence properties of backpropagation," Tech. Rep. UMIACS-TR-96-22 and CS-TR-3617, Institute for Advanced Computer Studies, Univ. of Maryland, Apr. 1996.
30. W. S. Lee, P. L. Bartlett, and R. C. Williamson, "Efficient agnostic learning of neural networks with bounded fan-in," IEEE Trans. Inform. Theory, vol. 42, pp. 2118-2132, Nov. 1996.
31. G. G. Lorentz, Approximation of Functions. New York: Holt, Rinehart and Winston, 1966.
32. G. Loy and P. L. Bartlett, "Generalization and the size of the weights: An experimental study," in Proc. 8th Australian Conf. on Neural Networks, M. Dale, A. Kowalczyk, R. Slaviero, and J Szymanski, Eds., Telstra Res. Labs., 1997, pp. 60-64.
33. G. Lugosi and M. Pintér, "A data-dependent skeleton estimate for learning," in Proc. 9th Annu. Conf. on Computational Learning Theory, New York: ACM Press, 1996, pp. 51-56.
34. W. Maass, "Vapnik-Chervonenkis dimension of neural nets," in The Handbook of Brain Theory and Neural Networks, M. A. Arbib, Ed. Cambridge, MA: MIT Press, 1995, pp. 1000-1003.
35. D. Pollard, Convergence of Stochastic Processes. New York: Springer, 1984.
36. R. E. Schapire, Y. Freund, P. L. Bartlett, and W. S. Lee, "Boosting the margin: A new explanation for the effectiveness of voting methods," in Machine Learning: Proc. 14th Int. Conf., 1997.
37. J. Shawe-Taylor, P. L. Bartlett, R. C. Williamson, and M. Anthony, "A framework for structural risk minimisation," in Proc. 9th Annu. Conf. on Computational Learning Theory. New York: ACM Press, 1996, pp. 68-76.
38. _, "Structural risk minimization over data-dependent hierarchies, September 1996," Tech. Rep.
39. E. D. Sontag, "Feedforward nets for interpolation and classification," J. Comput. Syst. Sci., vol. 45, pp. 20-48, 1992.
40. V. N. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982.
41. V. N. Vapnik and A. Y. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory Probab. Its Applicat., vol. 16, no. 2, pp. 264-280, 1971.