Learning by canonical smooth estimation - Part II: Learning and choice of model complexity

IEEE Transactions on Automatic Control

Volumn 41, Issue 4, 1996, Pages 557-569

  Buescher, Kevin L ^a,b   Kumar, P R ^a,c  

^a IEEE

^b LOS ALAMOS NATIONAL LABORATORY   (United States)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ERRORS; MATHEMATICAL MODELS; OPTIMIZATION; PROBABILITY; RANDOM PROCESSES;

CANONICAL SMOOTH ESTIMATION; GENERALIZATION ERROR; MODEL COMPLEXITY;

PARAMETER ESTIMATION;

EID: 0030128525     PISSN: 00189286     EISSN: None     Source Type: Journal    
DOI: 10.1109/9.489276     Document Type: Article

References (48)

1. K. L. Buescher and P. R. Kumar, "Learning by canonical smooth estimation, Part I: Simultaneous estimation," IEEE Trans. Automat. Contr., vol. 41, no. 4, pp. 545-556, 1996.
2. V. N. Vapnik and A. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory Probability Its Appl., vol. 16, no. 2, pp. 264-280, 1971.
3. _, "Necessary and sufficient conditions for the uniform convergence of means to their expectations," Theory Probability Its Appl., vol. 26, no. 3, pp. 532-553, 1981.
4. V. N. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982.
5. A. N. Kolmogorov and V. M. Tihomirov, "∈-entropy and ∈-capacity of sets in functional spaces," Amer. Math. Soc. Translations, Ser. 2, vol. 17, pp. 277-364, 1961.
6. M. Stone, "Cross-validatory choice and assessment of statistical predictions," J. Royal Statistical Soc. B, vol. 36, pp. 111-147, 1974.
7. G. M. Benedek and A. Itai. "Learnability by fixed distributions," in Computational Learning Theory: Proceedings of the 1988 Workshop. San Mateo, CA: Morgan Kaufman, 1988, pp. 80-90.
8. S. R. Kulkarni, "On metric entropy, Vapnik-Chervonenkis dimension, and learnability for a class of distributions," Center for Intelligent Control Systems, M.I.T., Cambridge, MA, Tech. Rep. CICS-P-160, Oct. 1989.
9. _, "Problems of computational and information complexity in machine vision and learning," Center for Intelligent Control Systems, M.I.T., Cambridge, MA, Tech. Rep. CICS-TH-298, July 1991; also Ph.D. dissertation, Dep. Electrical Engineering and Computer Science.
10. S. R. Kulkarni and M. Vidyasagar, "Learning decision rules for pattern classification under a family of probability measures," preprint, Oct. 1993.
11. L. Devroye, "Automatic pattern recognition: A study of the probability of error," IEEE Trans. Pattern Analysis Machine Intell., vol. 10, no. 4, pp. 530-543, 1988.
12. S. Ben-David, G. M. Benedek, and Y. Mansour, "A parameterization scheme for classifying models of learnability," in Computational Learning Theory: Proceedings of the Second Workshop. San Mateo, CA: Morgan Kaufmann, 1989, pp. 285-302.
13. J. Rissanen, "A universal prior for integers and estimation by minimum description length," Annals Statistics, vol. 11, no. 2, pp. 416-431, 1983.
14. D. Haussler, "Decision theoretic generalizations of the PAC model for neural net and other learning applications," Inform. Computation, vol. 100, pp. 78-150, 1992.
15. A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth, "Learnability and the Vapnik-Chervonenkis dimension," J. ACM, vol. 36, no. 3, pp. 939-965, 1989.
16. R. M. Dudley, "Central limit theorems for empirical measures," Annals Probability, vol. 6, no. 6, pp. 899-929, 1978.
17. R. S. Wenocur and R. M. Dudley, "Some special Vapnik-Chervonenkis classes," Discrete Math., vol. 33, pp. 313-318, 1981.
18. R. M. Dudley, "A course on empirical processes," in Lecture Notes in Mathematics, vol. 1097. New York: Springer-Verlag, 1984.
19. E. Baum and D. Haussler, "What size net gives valid generalization?" Neural Computation, vol. 1, pp. 151-160, 1989.
20. A. Macintyre and E. D. Sontag, "Finiteness results for sigmoidal neural networks," in Proc. Twenty-Fifth Ann. ACM Symp. Theory Computing, pp. 325-334, Association for Computing Machinery, 1993.
21. P. W. Goldberg and M. R. Jerrum, "Bounding the Vapnik-Chervonenkis dimension of concept classes parameterized by real numbers," Submitted to Machine Learning;
22. also in Proc. Sixth Ann. ACM Conf. Computational Learning Theory, 1993.
23. W. Maass, "Bounds for the computational power and learning complexity of analog neural nets," in Proc. Twenty-Fifth Ann. ACM Symp. Theory Computing, pp. 335-344, Association for Computing Machinery, 1993.
24. D. Pollard, Convergence of Stochastic Processes. New York: Springer-Verlag, 1984.
25. _, Empirical Processes: Theory and Applications. Hayward, CA: Institute of Mathematical Statistics, 1990.
26. D. Haussler, "Generalizing the PAC model: Sample size bounds from metric dimension based uniform convergence results," in 30th Annu. IEEE Symp. Foundations Computer Sci., 1989, pp. 40-45.
27. U. Grenander, Abstract Inference. New York: Wiley, 1981.
28. D. D. Cox, "Approximation of least squares regression on nested subspaces," Annals Statistics, vol. 16, no. 2, pp. 713-732, 1988.
29. V. N. Vapnik and A. Chervonenkis, "Minimization of expected risk based on empirical data," in Proc. First World Congr. Bernoulli Soc., vol. 2. Utrecht, The Netherlands, pp. 821-832, VNU Science Press, 1987.
30. V. N. Vapnik, "Inductive principles of the search for empirical dependences (methods based on weak convergence of probability measures)," in Computational Learning Theory: Proceedings of the Second Workshop. San Mateo, CA: Morgan Kaufmann, 1989, pp. 3-21.
31. H. White, "Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings," Neural Networks, vol. 3, no. 5, pp. 535-549, 1990.
32. G. Lugosi and K. Zeger, "Nonparametric estimation via empirical risk minimization," IEEE Trans. Inform. Theory, vol. 41, no. 3, pp. 677-687, 1995.
33. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, no. 6, pp. 716-723, 1974.
34. V. N. Vapnik and A. Chervonenkis, "Ordered risk minimization, I," Automation Remote Contr., vol. 35, pp. 1226-1235, Aug. 1974.
35. _, "Ordered risk minimization, II," Automation Remote Contr., vol. 35, pp. 1403-1412, Sep. 1974.
36. G. Schwarz, "Estimating the dimension of a model," Annals Statistics, vol. 6, no. 2, pp. 461-464, 1978.
37. L, cross-validation and generalized cross-validation: Discrete index set," Annals Statistics, vol. 15, no. 3, pp. 958-975, 1987.
38. A. R. Barron, "Complexity regularization with applications to artificial neural networks," in Proceedings of the NATO ASI on Nonparametric Functional Estimation. Norwood, MA: Kluwer, 1990.
39. _, "Approximation and estimation bounds for artificial neural networks," in Computational Learning Theory: Proceedings of the Fourth Annual Workshop. San Mateo, CA: Morgan Kaufmann, 1991, pp. 243-249.
40. A. R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, no. 4, pp. 1034-1054, 1991.
41. S. M. Weiss and C. A. Kulikowski, Computer Systems That Learn. San Mateo, CA: Morgan Kaufmann, 1991.
42. B. Efron, "Estimating the error rate of a prediction rule: Improvement on cross-validation," J. Amer. Statistical Association, vol. 78, pp. 316-331, June 1983.
43. D. Haussler, "Quantifying inductive bias: AI learning algorithms and Valiant's learning framework," Artificial Intelligence, vol. 36, no. 2, pp. 117-222, 1988.
44. K. L. Buescher, "Learning and smooth simultaneous estimation of errors based on empirical data," Coordinated Science Lab. Rep. DC-144, Univ. Illinois, Urbana, IL, Tech. UILU-ENG-92-2246, Nov. 1992; also Ph.D. dissertation, Dep. Electrical and Computer Engineering.
45. D. Aldous and U. Vazirani, "A Markovian extension of Valiant's learning model," in 31st Ann. IEEE Symp. Foundations Computer Sci., Oct. 1990, pp. 392-404.
46. A. B. Nobel, "Evaluating the performance of a simple inductive procedure in the presence of overfilling error," in Computational Learning Theory: Proceedings of the Fourth Annual Workshop. San Mateo, CA, Morgan Kaufmann, 1991, pp. 267-274.
47. _, "On uniform laws of averages," Stanford University, Dep. Statistics, Stanford, CA, Tech. Rep. 81, June 1992; also Ph.D. dissertation, Dep. Electrical Engineering.
48. M. Anthony and N. Biggs, Computational Learning Theory. Cambridge, MA: Cambridge Univ. Press, 1992.