Distribution Estimation Consistent in Total Variation and in Two Types of Information Divergence

IEEE Transactions on Information Theory

Volumn 38, Issue 5, 1992, Pages 1437-1454

  Barron, Andrew R ^a   Györfi, László ^b   van der Meulen, Edward C ^c  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b BUDAPEST UNIVERSITY OF TECHNOLOGY AND ECONOMICS   (Hungary)

^c UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Consistent distribution estimation; histogram based estimate; information divergence; reversed order information divergence; total variation

Indexed keywords

PROBABILITY;

CONSISTENT DISTRIBUTION ESTIMATION; HISTOGRAM-BASED ESTIMATORS; INFORMATION DIVERGENCE; NONPARAMETRIC ESTIMATION; REVERSE-ORDER INFORMATION DIVERGENCE; TOTAL VARIATION;

INFORMATION THEORY;

EID: 0026914305     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/18.149496     Document Type: Article

References (26)

1. S. Abou-Jaoude, “Sur une condition nécessaire et suffisante de L1-convergence presque complète de l'estimateur de la partition fixe pour une densité,” Comptes Rendus de I'Académie des Sciences de Paris Série A, t. 283, pp. 1107–1110, 1976.
2. “Conditions nécessaires et suffisantes de convergence L1 enprobabilité de l'histogramme pour une densité”, Annates de Vlnstitut Henri Poincaré, vol. XII, pp. 213–231, 1976.
3. A. R. Barron, “The convergence in information of probability density estimators,” presented at IEEE Int. Symp Inform. Theory, Kobe, Japan, June 19-24, 1988.
4. “Uniformly powerful goodness of fit tests,” Ann. Statist., vol. 17, pp. 107–124, Mar. 1989.
5. A. R. Barron and T. M. Cover, “A bound on the financial value of information,” IEEE Trans. Inform. Theory, vol. 34, pp. 1097–1100, Sept. 1988.
6. A. R. Barron and C.-H. Sheu, “Approximation of density functions by sequences of exponential families,” Ann. Statist., vol. 19, pp. 1347–1369, Sept. 1991.
7. B. S. Clarke and A. R. Barron, “Information-theoretic asymptotics of Bayes methods,” IEEE Trans. Inform. Theory, vol. 36, pp. 453–471, May 1990.
8. T. M. Cover, “The admissibility properties of Gilbert's encoding for unknown source probabilities,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 216–217, Jan. 1972.
9. I. Csiszar, “Information-type measures of difference of probability distributions and indirect observations,” Studia Sci. Math. Hung., vol. 2, pp. 299–318, 1967.
10. “Generalized entropy and quantization problems,” in Trans. Sixth Prague Conf. Information Theory, Statistical Decision Functions, Random Processes. Prague: Academia, 1973, pp. 159–174.
11. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. Budapest: Akadémiai Kiadó, 1981.
12. L. D. Davisson, “Universal noiseless coding,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 783–795, Nov. 1973.
13. L. Devroye, A Course in Density Estimation. Boston: Birkhäuser, 1987.
14. L. Devroye and L. Gyorfi, Nonparametric Density Estimation: The L, View. New York: Wiley, 1985.
15. “No empirical probability measure can converge in total variation sense for all distributions,” Ann. Statist., vol. 18, pp. 1496–1499, May 1990.
16. T. S. Ferguson, “A Bayesian analysis of some nonparametric problems,” Ann. Statist., vol. 1, pp. 209–230, Jan. 1973.
17. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
18. E. N. Gilbert, “Codes based on inaccurate source probabilities,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 304–314, May 1971.
19. L. Gyorfi and E. C. van der Meulen, “Density-free convergence properties of various estimators of entropy,” Comput. Statist. Data Anal., vol. 5, pp. 425–436, 1987.
20. “Density estimation a.s. consistent in information divergence,” in Proc. Fourth Joint Swedish-Soviet Int. Workshop Inform. Theory, Gotland, Sweden, Aug. 27-Sept. 1, 1989, pp. 112–116.
21. W. Hoeffding, “Probability inequalities for sums of bounded random variables,” J. Amer. Statist. Assoc., vol. 58, pp. 13–30, Mar. 1963.
22. “Asymptotically optimal tests for multinomial distributions,” Ann. Math. Statist., vol. 36, pp. 369–401, Apr. 1965.
23. J. H. B. Kemperman, “On the optimum rate of transmitting information,” Ann. Math. Statist., vol. 40, pp. 2156–2177, Dec. 1969.
24. R. E. Krichevsky and V. K. Trofimov, “The performance of universal encoding,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 199–207, Mar. 1981.
25. S. Kullback, “A lower bound for discrimination in terms of variation,” IEEE Trans. Inform. Theory, vol. IT-13, pp. 126–127, Jan. 1967.
26. J. Rissanen, T. P. Speed, and B. Yu, “Density estimation by stochastic complexity,” in IEEE Trans. Inform. Theory, vol. 38, pp. 315–323, Mar. 1992.