Nonparametric Estimation via Empirical Risk Minimization

IEEE Transactions on Information Theory

Volumn 41, Issue 3, 1995, Pages 677-687

  Zeger, Kenneth ^b  

^a BUDAPEST UNIVERSITY OF TECHNOLOGY AND ECONOMICS   (Hungary)

^b UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

consistency; neural networks; nonparametric estimation; pattern recognition; Regression estimation; series methods; sieves

Indexed keywords

CURVE FITTING; ERRORS; ESTIMATION; NEURAL NETWORKS; PATTERN RECOGNITION; REGRESSION ANALYSIS;

CONSISTENCY; ESTIMATORS; NONPARAMETRIC ESTIMATION; REGRESSION ESTIMATION; SERIES METHODS;

INFORMATION THEORY;

EID: 0029307575     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/18.382014     Document Type: Article

References (78)

1. M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, “The method of potential functions for the problem of restoring the characteristic of a function converter from randomly observed points,” Automat. Remote Contr., vol. 25, pp. 1546–1556, 1964.
2. —, “The probability problem of pattern recognition learning and the method of potential functions,” Automat. Remote Contr., vol. 25, pp. 1307–1323, 1964.
3. —, “Theoretical foundations of the potential function method in pattern recognition learning,” Automat. Remote Contr., vol. 25, pp. 917–936, 1964.
4. A. R. Barron, “Statistical properties of artificial neural networks,” in Proc. 28th Conf. on Decision and Control (Tampa, FL, 1989.
5. —, “Approximation and estimation errors for artificial neural networks,” in Computational Learning Theory: Proc. 4th Annual Workshop. Morgan Kaufman, 1991.
6. —, “Complexity regularization with application to artificial neural networks,” in G. Roussas, Ed., Nonparametric Functional Estimation and Related Topics (NATO ASI Series). Dordrecht, The Netherlands: Kluwer, 1991, pp. 561–576.
7. —, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Trans. Inform. Theory, vol. 39, pp. 930–944, 1993.
8. —, “Approximation and estimation bounds for artificial neural networks,” Machine Learning, vol. 14, pp. 115–133, 1994.
9. A. R. Barron and T. M. Cover, “Minimum complexity density estimation,” IEEE Trans. Inform. Theory, vol. 37, pp. 1034–1054, 1991.
10. L. Birgé and P. Massart, “Rates of convergence for minimum contrast estimators,” Prob. Theory Related Fields, vol. 97, pp. 113–150, 1993.
11. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, “Learn-ability and the Vapnik—Chervonenkis dimension,” J. ACM, vol. 36, pp. 929–965, 1989.
12. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. Belmont, CA: Wadsworth Int., 1984.
13. K. L. Buescher and P. R. Kumar, “Learning by canonical smooth estimation, Part I: Simultanious estimation,” submitted to IEEE Trans. Automat. Contr., 1994.
14. —, “Learning by canonical smooth estimation, Part II: Learning and choice of model complexity,” submitted to IEEE Trans. Automat. Contr., 1994.
15. N. N. Cencov, “Evaluation of an unknown distribution density from observations,” Sov. Math.—Dokl, vol. 3, pp. 1559–1562, 1962.
16. T. M. Cover, “Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition,” IEEE Trans. Electron. Comput., vol. EC-14, pp. 326–334, 1965.
17. D. D. Cox, “Approximation of least squares regression on nested subspaces,” Annals Statist., vol. 16, pp. 713–732, 1988.
18. G. Cybenko, “Approximations by superpositions of sigmoidal functions,” Math. Contr., Signals, Syst., vol. 2, pp. 303–314, 1989.
19. L. Devroye, “Any discrimination rule can have an arbitrarily bad probability of error for finite sample size,” IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-4, pp. 154–157, 1982.
20. L. Devroye, “Automatic pattern recognition: A study of the probability of error,” IEEE Trans. Pattern Anal. Machine Intell., vol. 10, pp. 530–543, 1988.
21. L. Devroye and L. Györfi, “Distribution-free exponential bound on the L1 error of partitioning estimates of a regression function,” in F. Konecny, J. Mogyoródi, and W. Wertz, Eds., Proc. 4th Pannonian Symp. on Mathematical Statistics. Budapest, Hungary: Akadémiai Kiadó, 1983, pp. 67–76.
22. L. Devroye, L. Györfi, A. Krzyzak, and G. Lugosi, “On the strong universal consistency of nearest neighbor regression function estimates,” to appear in theAnnals Stat., Sept. 1994.
23. L. Devroye and A. Krzyzak, “An equivalence theorem for L1 convergence of the kernel regression estimate,” J. Statist. Planning and Inference, vol. 23, pp. 71–82, 1989.
24. L. Devroye and T. J. Wagner, “Nonparametric discrimination and density estimation,” Tech. Rep. 183, Electron. Res. Cen., Univ. of Texas, 1976.
25. —, “Distribution-free consistency results in nonparametric discrimination and regression function estimation,” Annals Statist., vol. 8, pp. 231–239, 1980.
26. R. M. Dudley, “Central limit theorems for empirical measures,” Annals Probab., vol. 6, pp. 899–929, 1978.
27. —, “Universal Donsker classes and metric entropy,” Annals Probab., vol. 15, pp. 1306–1326, 1987.
28. V. Fabian, “Stochastic approximation,” in J. S. Rustagi, Ed., Optimizing Methods in Statistics. New York, London: Academic Press, 1971, pp. 439–470.
29. —, “On neural network models and stochastic approximation,” preprint, 1992.
30. A. Faragó and G. Lugosi,” “Strong universal consistency of neural network classifiers,” IEEE Trans. Inform. Theory, vol. 39, pp. 1146–1151, 1993.
31. K. Funahashi, “On the approximate realization of continuous mappings by neural networks,” Neural Net., vol. 2, pp. 183–192, 1989.
32. A. R. Gallant, Nonlinear Statistical Models. New York: Wiley, 1987.
33. S. Geman and C. R. Hwang, “Nonparametric maximum likelihood estimation by the method of sieves,” Annals Statist., vol. 10, pp. 401–414, 1982.
34. N. Glick, “Sample-based multinomial classification,” Biometrics, vol. 29, pp. 241–256, 1973.
35. W. Greblicki, “Asymptotic efficiency of classifying procedures using the Hermite series estimate of multivariate probability densities,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 364–366, 1981.
36. W. Greblicki and M. Pawlak, “Classification using the Fourier series estimate of multivariate density functions,” IEEE Trans. Syst., Man, Cybern., vol. SMC-11, pp. 726–730, 1981.
37. —, “A classification procedure using the multiple Fourier series,” Inform. Sci., vol. 26, pp. 115–126, 1982.
38. —, “Almost sure convergence of classification procedures using Hermite series density estimates,” Pattern Recogn. Lett., vol. 2, pp. 13–17, 1983.
39. U. Grenander, Abstract Inference. New York: Wiley, 1981.
40. L. Györfi, “An upper bound of error probabilities for multihypothesis testing and its application in adaptive pattern recognition,” Probl. Contr. and Inform. Theory, vol. 5, pp. 449–457, 1975.
41. —, “Adaptive linear procedures under general conditions,” IEEE Trans. Inform. Theory, vol. IT-30, pp. 262–267, 1984.
42. —, “Universal consistencies of a regression estimate for unbounded regression functions,” in G. Roussas, Ed., Nonparametric Functional Estimation and Related Topics (NATO ASI Series). Dordrecht, The Netherlands: Kluwer, 1991, pp. 329–338.
43. W. Härdie, Applied Nonparametric Regression. Cambridge, UK: Cambridge Univ. Press, 1990.
44. D. Haussler, “Decision theoretic generalizations of the PAC model for neural net and other learning applications,” Inform, and Comput., vol. 100, pp. 78–150, 1992.
45. K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Net., vol. 4, pp. 251–257, 1991.
46. —, “Some new results on neural network approximation,” Neural Net., vol. 6, pp. 1069–1072, 1993.
47. K. Hornik, M. Stinchcombe, and H. White, “Multi-layer feedforward networks are universal approximators,” Neural Net., vol. 2, pp. 359–366, 1989.
48. A. N. Kolmogorov and V. M. Tikhomirov, “ε-entropy and ε-capacity of sets in function spaces,” Transi. Amer. Math. Soc., vol. 17, pp. 277–364, 1961.
49. R. A. Kronmal and M. E. Tarter, “The estimation of probability densities and cumulatives by Fourier series methods,” J. Amer. Statist. Assoc., vol. 63, pp. 925–952, 1968.
50. L. Ljung, G. Pflug, and H. Walk, Stochastic Approximation and Optimization of Random Systems. Basel, Boston, Berlin: Birkhäuser, 1992.
51. J. Mielniczuk and J. Tyrcha, “Consistency of multilayer perceptron regression estimators,” Neural Net., to appear, 1993.
52. A. S. Nemirovskiy, B. T. Polyak, and A. B. Tsybako, “Rate of convergence of nonparametric estimators of maximum-likelihood type,” Probl. Inform. Transmission, vol. 21, pp. 258–272, 1985.
53. A. S. Nemirovski, “Nonparametric estimation of smooth regression functions,” Eng. Cybern., vol. 23, no. 6, pp. 1–11, 1985.
54. A. B. Nobel, “On uniform laws of averages,” Ph.D. dissertation, Dep. Statist., Stanford Univ., Stanford, CA, 1992.
55. D. Nolan and D. Pollard, “U-processes: Rates of convergence,” Annals Statist., vol. 15, pp. 780–799, 1987.
56. D. Pollard, Convergence of Stochastic Processes. New York: Springer-Verlag, 1984.
57. —, Empirical Processes: Theory and Applications (NSF-CBMS Regional Conference Series in Probability and Statistics, Hayward, CA, Alexandria, VA, 1990).
58. H. Robbins and S. Monro, “A stochastic approximation method,” Annals Math. Stat., vol. 22, pp. 400–407, 1951.
59. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in D. E. Rumelhart, J. L. McCelland, and the PDP Research Group, Eds., Parallel Distributed Processing, Vol. I. Cambridge, MA: M.I.T. Press, 1986.
60. N. Sauer, “On the density of families of sets,” J. Combinatorial Theory Ser. A, vol. 13, pp. 145–147, 1972.
61. S. C. Schwartz, “Estimation of probability density by an orthogonal series,” Annals Math. Stat., vol. 38, pp. 1261–1265, 1967.
62. X. Shen and W. H. Wong, “Convergence rate of sieve estimates,” to appear in the Annals Statist., vol. 22, pp. 580–615, June 1994.
63. D. F. Specht, “Series estimation of a probability density function,” Technometrics, vol. 13, pp. 409–424, 1971.
64. C. Spiegelman and J. Sacks, “Consistent window estimation in nonparametric regression,” Annals Stat., vol. 8, pp. 240–246, 1980.
65. C. J. Stone, “Consistent nonparametric regression,” Annals Stat., vol. 8, pp. 1348–1360, 1977.
66. M. E. Tarter and R. A. Kronmal, “On multivariate density estimates based on orthogonal expansions,” Annals Math. Stat., vol. 41, pp. 718–722, 1970.
67. L. G. Valiant, “A theory of the learnable,” Commun. ACM, vol. 27, pp. 1134–1142, 1984.
68. S. Van de Geer, “Estimating a regression function,” Annals Stat., vol. 18, pp. 907–924, 1990.
69. J. Van Ryzin, “Bayes risk consistency of classification procedures using density estimation,” Sankhya, ser. A, vol. 28, pp. 161–170, 1966.
70. V. N. Vapnik, Estimation of Dependencies Based on Empirical Data. New York: Springer-Verlag, 1982.
71. V. N. Vapnik and A. Ya. Chervonenkis, “On the uniform convergence of relative frequencies of events to their probabilities,” Theory Probab. and Applic., vol. 16, pp. 264–280, 1971.
72. —, Theory of Pattern Recognition. Moscow, USSR: Nauka, 1974 (in Russian); German translation: Theorie der Zeichenerkennung. Berlin, Germany: Akademie-Verlag, 1979.
73. —, “Necessary and sufficient conditions for the uniform convergence of means to their expectations,” Theory Prob, and Appl., vol. 26, pp. 821–832, 1981.
74. H. White, “Some asymptotic results for learning in single hidden-layer feedforward network models,” J. Amer. Statist. Assoc., vol. 84, pp. 1003–1013, 1989.
75. —, “Connectionist nonparametric regression: multilayer feedforward networks can learn arbitrary mappings,” Neural Net., vol. 3, pp. 535–549, 1990.
76. —, “Nonparametric estimation of conditional quantiles using neural networks,” in Proc. 23rd Symp. of the Interface: Computing Science and Statistics, 1991.
77. C. T. Wolverton and T. J. Wagner, “Asymptotically optimal discriminant functions for pattern classification,” IEEE Trans. Syst., Sci., Cybern., vol. 15, pp. 258–265, 1969.
78. W. H. Wong and X. Shen, “Probability inequalities for likelihood ratios and convergence rates of sieve MLE's, Tech. Rep. 346, Dep. Stat., University of Chicago, Chicago, IL, 1992.