Minimum Complexity Density Estimation

IEEE Transactions on Information Theory

Volumn 37, Issue 4, 1991, Pages 1034-1054

  Barron, Andrew R ^a   Cover, Thomas M ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

bounds on redundancy; consistency; density estimation; discovery of probability laws; Kolmogorov complexity; minimum description length criterion; model selection; resolvability of functions; statistical convergence rates; universal data compression

Indexed keywords

INFORMATION THEORY;

DENSITY ESTIMATORS; MINIMUM COMPLEXITY;

STATISTICAL METHODS;

EID: 0026190366     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/18.86996     Document Type: Article

References (54)

1. A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Probl. Peredach. Inform., vol. 1, pp. 3–11, 1965.
2. V. V. V'Yugin, “On the defect of randomness of a finite object with respect to measures with given complexity bounds,” Theory Probab. Appl., vol. 32, pp. 508–512, 1987.
3. T. M. Cover, “Generalization on patterns using Kolmogorov complexity,” in Proc. First Int. Joint Conf. Pattern Recog., Washington, DC, Oct. 1973.
4. ––, “Kolmogorov complexity, data compression, and inference,” in The Impact of Processing Techniques on Communications, J. K. Skwirzynski, Ed. Boston, MA: Martinus Nijhoff Publ., 1985, pp. 23–34.
5. T. M. Cover, P. Gacs, and R. M. Gray, “Kolmogorov's contributions to information theory and algorithmic complexity,” Ann. Probab., vol. 17, pp. 840–865, July 1989.
6. J. Rissanen, “Modeling by shortest data description,” Automatica, vol. 14, pp. 465–471, 1978.
7. ––, “A universal prior for integers and estimation by minimum description length,” Ann. Statist., vol. 11, pp. 416–431, June 1983.
8. ——, “Universal coding, information, prediction, and estimation,” IEEE Trans. Inform. Theory, vol. 30, pp. 629–636, July 1984.
9. ——, “Stochastic complexity and modeling,” Ann. Statist., vol. 14, pp. 1080–1100, Sept. 1986.
10. ——, “Stochastic complexity and sufficient statistics,” J. Roy. Statist. Soc. B, vol. 49, pp. 223–239, 1987.
11. ——, Stochastic Complexity in Statistical Inquiry. Teaneck, NJ: World Scientific Publ., 1989.
12. C. S. Wallace and D. M. Boulton, “An information measure for classification,” Comput. J., vol. 11, pp. 185–194, 1968.
13. C. S. Wallace and P. R. Freeman, “Estimation and inference by compact coding,” J. Roy. Statist. Soc. B, vol. 49, pp. 240–265, 1987.
14. R. Sorkin, “A quantitative OccaM's razor,” Int. J. Theoretic Phys., vol. 22, pp. 1091–1103, 1983.
15. A. R. Barron, “Convergence of logically simple estimates of unknown probability densities,” presented at IEEE Int. Symp. Inform. Theory, St. Jovite, Canada, Sept. 26–30, 1983.
16. ——, “Logically smooth density estimation,” Ph.D. dissertation, Dept. Elect. Eng., Stanford Univ., Stanford, CA, Aug. 1985.
17. ——, “Complexity regularization,” in Proceedings NATO Advanced Study Institute on Nonparametric Functional Estimation, G. Roussas, Ed. Dordrecht, The Netherlands: Kluwer Academic Publ., 1991.
18. T. M. Cover, “A hierarchy of probability density function estimates,” in Frontiers in Pattern Recognition. New York: Academic Press, 1972, pp. 83–98.
19. L. D. Davisson, “Universal noiseless coding,” IEEE Trans. Inform. Theory., vol. 19, pp. 783–795, Nov. 1973.
20. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
21. R. J. Solomonoff, “A formal theory of inductive inference,” Inform. Contr., vol. 7, pp. 224–254, 1964.
22. G. J. Chaitin, “On the length of programs for computing finite binary sequences,” J. Assoc. Comput. Machine, vol. 13, pp. 547–569, 1966.
23. ——, “A theory of program size formally identical to information theory, J. Assoc. Comput. Machine, vol. 22, pp. 329–340, 1975.
24. L. A. Levin, “On the notion of a random sequence,” Soviet Math. Dokl., vol. 14, pp. 1413–1416, 1973.
25. ——, “Laws of information conservation and aspects of the foundations of probability theories,” Probl. Inform. Transm., vol. 10, pp. 206–210, 1974.
26. B. S. Clarke and A. R. Barron, “Information theoretic asymptotics of Bayes methods,” IEEE Trans. Inform. Theory, vol. 36, no. 3, pp. 453–471, May 1990.
27. B. S. Clarke, “Asymptotic cumulative risk and Bayes risk under entropy loss, with applications.” Ph.D. dissertation, Dept. Statist., Univ. of Illinois, Urbana, IL, July 1989.
28. R. E. Krichevsky and V. K. Trofimov, “The performance of universal encodings,” IEEE Trans. Inform. Theory, vol. 27, pp. 199–207, Mar. 1981.
29. H. Jeffreys, Theory of Probability. Oxford: Oxford Univ. Press, 1967.
30. G. Schwarz, “Estimating the dimension of a model,” Ann. Statist., vol. 6, pp. 461–464, 1978.
31. A. R. Barron and C. Sheu, “Approximation of density functions by sequences of exponential families,” Ann. Statist., vol. 19, no. 3, Sept. 1991.
32. R. Shibata, “An optimal selection of regression variables,” Biometrika, vol. 68 pp. 45–54, 1981.
33. K. C. Li, “Asymptotic optimality for Cp, CL, cross-validation, and generalized cross-validation: Discrete index set,” Ann. Statist., vol. 15, pp. 958–975, Sept. 1987.
34. H. Akaike, “Information theory and an extension of the maximum likelihood principle,” in Proc. 2nd Int. Symp. Inform. Theory, P. N. Petrov and F. Csaki, Eds. Budapest: Akademia Kiado, 1983, pp. 267–281.
35. P. Hall and E. J. Hannan, “On stochastic complexity and nonparametric density estimation,” Biometrika, vol. 75, pp. 705–714, 1988.
36. B. Yu and T. P. Speed, “Stochastic complexity and model selection II: Histograms,” Tech. rep. no. 241, Dept, of Statist. Univ. of California, Berkeley, Mar. 1990.
37. A. N. Kolmogorov and V. M. Tihomirov, “e-entropy and e-capacity of sets in function spaces,” Uspehi, vol. 3, pp. 3–86, 1959.
38. M. S. Birman and M. Z. Solomjak, “Piecewise-polynomial approximations of functions of the classes [formula omitted]” Mat. USSR-Sbornik, vol. 2, pp. 295–317, 1967.
39. U. Granander, Abstract Inference. New York: Wiley, 1981.
40. J. Bretagnolle and C. Huber, “Estimation des densites: Risque minimax”, Z. Wahrscheninlichkeitstheorie verw. Gebiete, vol. 47, pp. 119–137, 1979.
41. S. Y. Efroimovich and M. S. Pinsker, “Estimation of square-inte-grable probability density of a random variable,” Probl. Inform. Transm., vol. 18, pp. 175–189, 1983.
42. Y. G. Yatracos, “Rates of convergence of minimum distance estimators and Kolmogorov's entropy,” Ann. Statist., vol. 13, pp. 768–774, June 1985.
43. A. R. Barron, “Uniformly powerful goodness of fit tests,” Ann. Statist., vol. 17, no. 1, Mar. 1989.
44. E. J. G. Pitman, Some Basic Theory for Statistical Inference. London: Chapman and Hall, 1979.
45. H. Chernoff, “A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations,” Ann. Math. Statist., vol. 23, pp. 493–507, 1952.
46. W. Hoeffding, “Probability inequalities for sums of bounded random variables,” J. Amer. Statist. Assoc., vol. 58, pp. 13–30, Mar. 1963.
47. D. D. Cox, “Approximation of least squares regression on nested subspaces,” Ann. Statist., vol. 16, pp. 713–732, June 1988.
48. A. R. Barron and R. L. Barron, “Statistical learning networks: A unifying view,” Computing Science and Statistics: Proceedings of the 20th Symposium on the Interface. Fairfax, Virginia, April 21–23, 1988, E. Wegman, D. T. Gantz, and J. J. Miller, Eds. Alexandria, VA: Amer. Statist. Assoc., 1988.
49. A. R. Barron, “Statistical properties of artificial neural networks,” presented at Proc. 28th IEEE Conf. Decision Contr., Tampa, Florida, Dec. 1989.
50. G. G. Lorentz, Approximation of Functions. New York: Holt, Rinehart, and Winston, 1966.
51. C. A. Rogers, “Lattice coverings of space,” Mathematika, vol. 6, pp. 33–39, 1959.
52. H. S. M. Coxeter, L. Few, and C. A. Rogers, “Covering space with equal spheres,” Mathematika, vol. 6, pp. 147–157, 1959.
53. J. H. Conway and N. J. A. Sloane, Sphere Packings, Lattices, and Groups. New York: Springer-Verlag, 1988.
54. A. R. Barron, L. Györfi, and E. C. van der Meulen, “Distribution estimation convergent in total variation and in informational divergence,” to appear in IEEE Trans. Inform. Theory.