A class of rényi information estimators for multidimensional densities

Annals of Statistics

Volumn 36, Issue 5, 2008, Pages 2153-2182

  Leonenko, Nikolai ^a   Pronzato, Luc ^b   Savani, Vippal ^a  

^a CARDIFF UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF NICE SOPHIA ANTIPOLIS   (France)

Author keywords

Entropy estimation; Estimation of divergence; Estimation of statistical distance; Havrda Charv t entropy; Nearest neighbor distances; R nyi entropy; Tsallis entropy

Indexed keywords

EID: 54349103517     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/07-AOS539     Document Type: Article

References (47)

1. ALEMANY, P. and ZANETTE, S. (1992). Fractal random walks from a variational formalism for Tsallis entropies. Phys. Rev. E 49 956-958.
2. BASSEVILLE, M. (1996). Information entropies, divergences et moyennes. Research Report IRISA nb. 1020.
3. BEIRLANT, J., DUDEWICZ, E. J., GYÖRFI, L. andVAN DER MEULEN, E. C. (1997). Nonparametric entropy estimation: An overview. Internat. J. Math. Statist. Sci. 6 17-39.
4. BICKEL, P. J. and BREIMAN, L. (1983). Sums of functions of nearest neighbor distances, moment bounds, limit theorems and a goodness of fit test. Ann. Probab. 11 185-214.
5. BIERENS, H. J. (1994). Topics in Advanced Econometrics. Cambridge Univ. Press.
6. BRONIATOWSKI, M. (2003). Estimation of the Kullback-Leibler divergence. Math. Methods Statist. 12 391-409.
7. DEVROYE, L. P. and WAGNER, T. J. (1977). The strong uniform consistency of nearest neighbor density estimates. Ann. Statist. 5 536-540.
8. EVANS, D., JONES, A. J. and SCHMIDT, W. M. (2002). Asymptotic moments of near-neighbour distance distributions. R. Soc. Lond. Proc. Ser. A Math. Phys. Eng. Sci. 458 2839-2849.
9. FELLER, W. (1966). An Introduction to Probability Theory and its Applications. II. Wiley, New York.
10. FRANK, T. and DAFFERTSHOFER, A. (2000). Exact time-dependent solutions of the Rényi Fokker-Planck equation and the Fokker-Planck equations related to the entropies proposed by Sharma and Mittal. Phys. A 285 351-366.
11. GORIA, M. N., LEONENKO, N. N., MERGEL, V. V. and NOVI INVERARDI, P. L. (2005). A new class of random vector entropy estimators and its applications in testing statistical hypotheses. J. Nonparametr. Statist. 17 277-297.
12. HALL, P. (1986). On powerful distributional tests based on sample spacings. J. Multivariate Statist. 19 201-225.
13. HALL, P. and MORTON, S. (1993). On the estimation of entropy. Ann. Inst. Statist. Math. 45 69-88.
14. HALL, P., PARK, B. and SAMWORTH, R. (2004). Choice of neighbour order in nearest-neighbour classification. Manuscript.
15. HAVRDA, J. and CHARVÁT, F. (1967). Quantification method of classification processes. Concept of structural α-entropy. Kybernetika (Prague) 3 30-35.
16. HERO, III, A. O., MA, B., MICHEL, O. and GORMAN, J. (2002). Applications of entropic spanning graphs. IEEE Signal. Proc. Magazine 19 85-95. (Special Issue on Mathematics in Imaging.)
17. HERO, III, A. O. and MICHEL, O. J. J. (1999). Asymptotic theory of greedy approximations to minimal k-point random graphs. IEEE Trans. Inform. Theory 45 1921-1938.
18. HEYDE, C. C. and LEONENKO, N. N. (2005). Student processes. Adv. in Appl. Probab. 37 342-365.
19. JIMÉNEZ, R. and YUKICH, J. E. (2002). Asymptotics for statistical distances based on Voronoi tessellations. J. Theoret. Probab. 15 503-541.
20. KAPUR, J. N. (1989). Maximum-Entropy Models in Science and Engineering. Wiley, New York.
21. KOZACHENKO, L. and LEONENKO, N. (1987). On statistical estimation of entropy of a random vector. Problems Inform. Transmission 23 95-101.
22. [Translated from Prolemy Predachi Informatsii 23 (1987) 9-16 (in Russian).]
23. KRASKOV, A., STÖGBAUER, H. and GRASSBERGER, P. (2004). Estimating mutual information. Phys. Rev. E 69 1-16.
24. LEARNED-MILLER, E. and FISHER, J. (2003). ICA using spacings estimates of entropy. J. Machine Learning Research 4 1271-1295.
25. LIERO, H. (1993). A note on the asymptotic behaviour of the distance of the kth nearest neighbour. Statistics 24 235-243.
26. LOÈVE, M. (1977). Probability Theory I, 4th ed. Springer, New York.
27. LOFTSGAARDEN, D. O. and QUESENBERRY, C. P. (1965). A nonparametric estimate of a multivariate density function. Ann. Math. Statist. 36 1049-1051.
28. MILLER, E. (2003). A new class of entropy estimators for multidimensional densities. In Proc. ICASSP'2003.
29. MOORE, D. S. and YACKEL, J. W. (1977). Consistency properties of nearest neighbor density function estimators. Ann. Statist. 5 143-154.
30. NEEMUCHWALA, H., HERO, A. and CARSON, P. (2005). Image matching using alpha-entropy measures and entropic graphs. Signal Processing 85 277-296.
31. PENROSE, M. D. (2000). Central limit theorems for k-nearest neighbour distances. Stochastic Process. Appl. 85 295-320.
32. PRONZATO, L., THIERRY, É. and WOLSZTYNSKI, É. (2004). Minimum entropy estimation in semi-parametric models: A candidate for adaptive estimation? In mODa 7 - Advances in Model-Oriented Design and Analysis. Contrib. Statist. 125-132. Physica, Heidelberg.
33. REDMOND, C. and YUKICH, J. E. (1996). Asymptotics for Euclidean functionals with power-weighted edges. Stochastic Process. Appl. 61 289-304.
34. RÉNYI, A. (1961). On measures of entropy and information. Proc. 4th Berkeley Sympos. Math. Statist. Probab. I 547-561. Univ. California Press, Berkeley.
35. SCOTT, D. W. (1992). Multivariate Density Estimation. Wiley, New York.
36. SONG, K. (2000). Limit theorems for nonparametric sample entropy estimators. Statist. Probab. Lett. 49 9-18.
37. SONG, K. (2001). Rényi information, loglikelihood and an intrinsic distribution measure. J. Statist. Plann. Inference 93 51-69.
38. TSALLIS, C. (1988). Possible generalization of Boltzmann-Gibbs statistics. J. Statist. Phys. 52 479-487.
39. TSALLIS, C. and BUKMAN, D. (1996). Anomalous diffusion in the presence of external forces: Exact time-dependent solutions and their thermostatistical basis. Phys. Rev. E 54 2197-2200.
40. TSYBAKOV, A. B. and VAN DER MEULEN, E. C. (1996). Root-n consistent estimators of entropy for densities with unbounded support. Scand. J. Statist. 23 75-83.
41. VAN ES, B. (1992). Estimating functionals related to a density by a class of statistics based on spacings. Scand. J. Statist. 19 61-72.
42. VASICEK, O. (1976). A test for normality based on sample entropy. J. Roy. Statist. Soc. Ser. B 38 54-59.
43. VICTOR, J. (2002). Binless strategies for estimation of information from neural data. Phys. Rev. E 66 1-15.
44. VIGNAT, C., HERO, III, A. O. and COSTA, J. A. (2004). About closedness by convolution of the Tsallis maximizers. Phys. A 340 147-152.
45. VIOLA, P. and WELLS, W. (1995). Alignment by maximization of mutual information. In 5th IEEE Internat. Conference on Computer Vision 16-23. Cambridge, MA.
46. WOLSZTYNSKI, É., THIERRY, É. and PRONZATO, L. (2005). Minimum entropy estimation in semi-parametric models. Signal Processing 85 937-949.
47. ZOGRAFOS, K. (1999). On maximum entropy characterization of Pearson's type II and VII multivariate distributions. J. Multivariate Anal. 71 67-75.