On the Relationship between Generalization Error, Hypothesis Complexity, and Sample Complexity for Radial Basis Functions

Neural Computation

Volumn 8, Issue 4, 1996, Pages 819-842

  Niyogi, Partha ^a   Girosi, Federico ^a  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0000482137     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/neco.1996.8.4.819     Document Type: Article

References (48)

1. Angluin, D. 1988. Queries and concept learning. Mach. Learn. 2, 319-342.
2. Barron, A. 1993. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39(3), 930-945.
3. Barron, A. 1994. Approximation and estimation bounds for artificial neural networks. Mach. Learn. 14, 115-133.
4. Barron, A., and Cover, T. 1991. Minimum complexity density estimation. IEEE Trans. Theory 37(4).
5. Baum, E. B., and Haussler, D. 1989. What size net gives valid generalization? Neural Comp. 1, 151-160.
6. Blum, A., and Rivest, R. L. 1988. Training a three-neuron neural net is NP-complete. In Proceedings of the 1988 Workshop on Computational Learning Theory, pp. 9-18. Morgan Kaufmann, San Mateo, CA.
7. Botros, S., and Atkeson, C. G. 1991. Generalization properties of Radial Basis Functions. In Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. Touretzky, eds., pp. 707-713. Morgan Kaufmann, San Mateo, CA.
8. Cohn, D., and Tesauro, G. 1991. Can neural networks do better than the VC bounds? In Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. Touretzky, eds., pp. 911-917. Morgan Kaufmann, San Mateo, CA.
9. Craven, P., and Wahba, G. 1979. Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross validation. Numer. Math. 31, 377-403.
10. Cybenko, G. 1989. Approximation by superposition of a sigmoidal function. Math. Control Syst. Signals 2(4), 303-314.
11. Devroye, L. 1981. On the almost everywhere convergence of nonparametric regression function estimate. Ann. Statist. 9, 1310-1319.
12. Dudley, R. M. 1987. Universal Donsker classes and metric entropy. Ann. Prob. 14(4), 1306-1326.
13. Dudley, R. M. 1989. Real Analysis and Probability, Mathematics Series. Wadsworth and Brooks/Cole, Pacific Grove, CA.
14. Efron, B. 1982. The Jacknife, the Bootstrap, and Other Resampling Plans. SIAM, Philadelphia.
15. Geman, S., Bienenstock, E., and Doursat, R. 1992. Neural networks and the bias/variance dilemma. Neural Comp. 4, 1-58.
16. Girosi, F. 1994. Regularization theory, Radial Basis Functions and networks. In From Statistics to Neural Networks. Theory and Pattern Recognition Applications, V. Cherkassky, J. H. Friedman, and H. Wechsler, eds. Subseries F, Computer and Systems Sciences, Springer-Verlag, Berlin.
17. Girosi, F., and Anzellotti, G. 1993. Rates of convergence for Radial Basis Functions and neural networks. In Artificial Neural Networks for Speech and Vision, R. J. Mammone, ed., pp. 97-113. Chapman & Hall, London.
18. Girosi, F., Jones, M., and Poggio, T. 1995. Regularization theory and neural networks architectures. Neural Comp. 7, 219-269.
19. Gish, H. 1990. A probabilistic approach to the understanding and training of neural network classifiers. In Proceedings of the ICASSP-90, pp. 1361-1365, Albuquerque, NM.
20. Hampshire, J. B. II, and Pearlmutter, B. A. 1990. Equivalence proofs for multilayer perceptron classifiers and the bayesian discriminant function. In Proceedings of the 1990 Connectionist Models Summer School, J. Elman, D. Touretzky, and G. Hinton, eds. Morgan Kaufmann, San Mateo, CA.
21. Haussler, D. 1992. Decision-theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation 100(1), 78-150.
22. Hornik, K., Stinchcombe, M., and White, H. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2, 359-366.
23. Ji, C., and Psaltis, D. 1992. The VC dimension versus the statistical capacity of multilayer networks. In Advances in Neural Information Processing Systems 4, S. J. Hanson, J. Moody, and R. P. Lippman, eds., Morgan Kaufmann, San Mateo, CA.
24. Jones, L. K. 1992. A simple lemma on greedy approximation in Hilbert space and convergence rates for Projection Pursuit Regression and neural network training. Ann. Statist. 20(1), 608-613.
25. Judd, S. 1988. Neural network design and complexity of learning. Ph.D. thesis, University of Massachusetts, Amherst, Amherst, MA.
26. Krzyzak, A. 1986. The rates of convergence of kernel regression estimates and classification rules. IEEE Trans. Inform. Theory IT-32(5), 668-679.
27. Levin, E., Tishby, N., and Solla, S. A. 1990. A statistical approach to learning and generalization in layered neural networks. Proc. IEEE 78(10), 1568-1574.
28. Lippmann, R. P. 1987. An introduction to computing with neural nets. IEEE ASSP Mag. April, 4-22.
29. Lorentz, G. G. 1986. Approximation of Functions. Chelsea Publishing, New York.
30. Mhaskar, H. N. 1993. Approximation properties of a multilayered feedforward artificial neural network. Adv. Comput. Math. 1, 61-80.
31. Mhaskar, H. N., and Micchelli, C. A. 1992. Approximation by superposition of a sigmoidal function. Adv. Appl. Math. 13, 350-373.
32. Moody, J. 1991. The effective number of parameters: An analysis of generalization and regularization in non-linear learning systems. In Advances in Neural Information Processing Systems 4, S. J. Hanson, J. Moody, and R. P. Lippman, eds., pp. 847-854. Morgan Kaufmann, San Mateo, CA.
33. Moody, J., and Darken, C. 1989. Fast learning in networks of locally-tuned processing units. Neural Comp. 1(2), 281-294.
34. Niyogi, P. 1995. The informational complexity of learning from examples. Ph.D. thesis, MIT, Cambridge, MA.
35. Niyogi, P., and Girosi, F. 1994. On the relationship between generalization error, hypothesis complexity, and sample complexity for radial basis functions. A.I. Memo 1467, Massachussetts Institute of Technology, 1994. URL ftp://publications.ai.mit.edu/ai-publications/1000-1499/AIM-1467.ps.Z.
36. Poggio, T., and Girosi, F. 1990. Networks for approximation and learning. Proc. IEEE 78(9).
37. Pollard, D. 1984. Convergence of Stochastic Processes. Springer-Verlag, Berlin.
38. Powell, M. J. D. 1992. The theory of radial basis functions approximation in 1990. In Advances in Numerical Analysis Volume II: Wavelets, Subdivision Algorithms and Radial Basis Functions, W. A. Light, ed., pp. 105-210. Oxford University Press, Oxford, England.
39. Richard, M. D., and Lippman, R. P. 1991. Neural network classifier estimates bayesian a-posteriori probabilities. Neural Comp. 3, 461-483.
40. Rissanen, J. 1989. Stochastic Complexity in Statistical Inquiry. World Scientific, Singapore.
41. Stein, E. M. 1970. Singular Integrals and Differentiability Properties of Functions. Princeton University Press, Princeton, NJ.
42. Tikhonov, A. N. 1963. Solution of incorrectly formulated problems and the regularization method. Soviet Math. Dokl. 4, 1035-1038.
43. Vapnik, V. N. 1982. Estimation of Dependencies Based on Empirical Data. Springer-Verlag, Berlin.
44. Vapnik, V. N., and Chervonenkis, A. Y. 1971. On the uniform convergence of relative frequencies of events to their probabilities. Th. Prob. Appl. 17(2), 264-280.
45. Vapnik, V. N., and Chervonenkis, A. Y. 1991. The necessary and sufficient conditions for consistency in the empirical risk minimization method. Patt. Recogn. Image Analysis 1(3), 283-305.
46. Wahba, G. 1990. Spline Models for Observational Data, Series in Applied Mathematics, Vol. 59. SIAM, Philadelphia.
47. Weigand, A. S., Rumelhart, D. E., and Huberman, B. A. 1991. Generalization by weight elimination with applications to forecasting. In Advances in Neural Information Processing Systems 3, R. Lippmann, J. Moody, and D. Touretzky, eds., Morgan Kaufmann, San Mateo, CA.
48. White, H. 1990. Connectionist nonparametric regression: Multilayer perceptrons can learn arbitrary mappings. Neural Networks 3, 535-549.