Divergence estimation of continuous distributions based on data-dependent partitions

IEEE Transactions on Information Theory

Volumn 51, Issue 9, 2005, Pages 3064-3074

  Wang, Qing ^a   Kulkarni, Sanjeev R ^a   Verdú, Sergio ^a  

^a Princeton University   (United States)

Author keywords

Bias correction; Data dependent partition; Divergence; Radon Nikodym derivative; Stationary and ergodic data; Universal estimation of information measures

Indexed keywords

ADAPTIVE ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; ESTIMATION; SET THEORY; THEOREM PROVING; TREES (MATHEMATICS);

BIAS CORRECTION; CONTINUOUS DISTRIBUTIONS; DATA-DEPENDENT PARTITIONS; DIVERGENCE ESTIMATION; INFORMATION MEASURES; RADON-NIKODYM DERIVATIVE; UNIVERSAL ESTIMATION;

INFORMATION THEORY;

EID: 26444495559     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.853314     Document Type: Article

References (18)

1. S. Kullback and R. A. Leibler, "On information and sufficiency," Ann. Math. Statist., vol. 22, no. 1, pp. 79-86, Mar. 1951.
2. P. J. Moreno, P. P. Ho, and N. Vasconcelos, "A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications," HP Laboratories, Cambridge, MA, Tech. Rep. HPL-2004-4, 2004.
3. P. Hall, "On Kullback-Leibler loss and density estimation," Ann. Statist., vol. 15, no. 4, pp. 1491-1519, Dec. 1987.
4. J. Beirlant, E. J. Dudewicz, L. Györfi, and E. C. van der Meulen, "Nonparametric entropy estimation: An overview," Int. J. Math. Statist. Sci., vol. 6, no. 1, pp. 17-39, 1997.
5. H. Cal, S. R. Kulkarni, and S. Verdú, "Universal estimation of entropye and divergence via block sorting," in Proc. IEEE Int. Syrup. Information Theory, Lausanne, Switzerland, Jun./Jul. 2002, p. 433.
6. H. Cai, S. R. Kulkarni, and S. Verdú, "Universal divergenceestimation for finite-alphabet sources," IEEE Trans. Inf. Theory, submitted for publication.
7. J. Ziv and N. Merhav, "A measure of relative entropy between individual sequences with application to universal classification," IEEE Trans. Inf. Theory, vol. 39, no. 4, pp. 1270-1279, Jul. 1993.
8. A. Hero, B. Ma, and O. Michel, "Estimation of Rényi information divergence via pruned minimal spanning trees," in IEEE Workshop on Higher Order Statistics, Caesaria, Israel, Jun. 1999.
9. G. A. Darbellay and I. Vajda, "Estimation of the information by an adaptive partitioning of the observation space," IEEE Trans. Inf. Theory, vol. 45, no. 4, pp. 1315-1321, May 1999.
10. G. Lugosi and A. Nobel, "Consistency of data-driven histogram methods for density estimation and classification," Ann. Statist., vol. 24, no. 2, pp. 687-706, 1996.
11. I. Gijbels and J. Mielniczuk, "Asymptotic properties of kernel estimators of the Radon-Nikodym derivative with applications to discriminant analysis," Statistica Sinica, vol. 5, pp. 261-278, 1995.
12. S. Panzeri and A. Treves, "Analytical estimates of limited sampling biases in different information measures," Network: Computation in Neural Systems, vol. 7, pp. 87-107, 1996.
13. E. Çinlar, Lecture Notes in Probability Theory. Unpublished.
14. L. Paninski, "Estimation of entropy and mutual information," Neural Comput., vol. 15, no. 6, pp. 1191-1253, Jun. 2003.
15. L. Devroye, A Course in Density Estimation, Boston, MA: Birkhäuser, 1987.
16. V. N. Vapnik and A. Ya. Chervonenkis, "On the uniform convergence of relative frequencies of events to their probabilities," Theory Probab. Its Applic., vol. 16, no. 2, pp. 264-280, 1971.
17. L. Devroye, L. Györfi, and G. Lugosi, A Probablistic Theory of Pattern Recognition. Berlin, Germany: Springer-Verlag, 1996.
18. T. M. Adams and A. B. Nobel, "On density estimation from ergodic processes," Ann. Probab., vol. 26, no. 2, pp. 794-804, Apr. 1971.