Estimation of nonlinear functionals of densities with confidence

IEEE Transactions on Information Theory

Volumn 58, Issue 7, 2012, Pages 4135-4159

  Sricharan, Kumar ^a   Raich, Raviv ^b   Hero III, Alfred O ^a  

^a UNIVERSITY OF MICHIGAN   (United States)

^b 1148 Kelley Engineering Center Corvallis   (United States)

Author keywords

Adaptive estimators; bias and variance tradeoff; bipartite k nearest neighbor ( NN) graphs; concentration bounds; convergence rates; data splitting estimators; entropy estimation

Indexed keywords

ADAPTIVE ESTIMATORS; BIAS AND VARIANCE; CONCENTRATION BOUNDS; CONVERGENCE RATES; DATA-SPLITTING ESTIMATORS; ENTROPY ESTIMATION; K-NEAREST NEIGHBORS;

MEAN SQUARE ERROR; PROBABILITY DENSITY FUNCTION; PROBABILITY DISTRIBUTIONS;

ESTIMATION;

EID: 84862560441     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2012.2195549     Document Type: Article

References (39)

1. A. O. Hero, B. Ma, O. Michel, and J. Gorman, "Applications of entropic spanning graphs," IEEE Signal Process. Mag., vol. 19, no. 5, pp. 85-95, Sep. 2002.
2. H. Neemuchwala and A. O. Hero, "Image registration in high dimensional feature space," presented at the presented at the SPIE Conf. Electron. Imag., San Jose, CA, Jan. 2005.
3. E. G. Miller and J. W. Fisher, III, "ICA using spacings estimates of entropy," in Proc. 4th Int. Symp. Independent Compon. Anal. Blind Signal Separation, 2003, pp. 1047-1052.
4. A. Lakhina, M. Crovella, and C. Diot, "Mining anomalies using traffic feature distributions," in Proc. ACM SIGCOMM, 2005, pp. 217-228.
5. A. Jain, "Image data compression: A review," Proc. IEEE, vol. 69, no. 3, pp. 349-389, Mar. 1981.
6. O. Vasicek, "A test for normality based on sample entropy," J. R. Statist. Soc. Series B (Methodol.), vol. 38, pp. 54-59, 1976.
7. E. J. Dudewicz and E. C. van der Meulen, "Entropy-based tests of uniformity," J. Amer. Statist. Assoc., vol. 76, pp. 967-974, 1981.
8. R. C. H. Cheng and N. A. K. Amin, "Estimating parameters in continuous univariate distributions with a shifted origin," J. R. Statist. Soc. Series B (Methodol.), vol. 11, pp. 394-403, 1983.
9. B. Ranneby, "The maximum spacing method. An estimation method related to the maximum likelihood method," Scand. J. Statist., vol. 11, pp. 93-112, 1984.
10. N. Leonenko, L. Prozanto, and V. Savani, "A class of Rényi information estimators for multidimensional densities," Ann. Statist., vol. 36, pp. 2153-2182, 2008.
11. A. O. Hero, J. Costa, and B. Ma, Asymptotic relations between minimal graphs and alpha-entropy Commun. Signal Process. Lab., Univ. Michigan, Tech. Rep. CSPL-334, Mar. 2003.
12. D. Pál, B. Póczos, and C. Szepesvári, "Estimation of Rényi entropy and mutual information based on generalized nearest-neighbor graphs," ArXiv E-Prints, Mar. 2010.
13. B. van Es, "Estimating functionals related to a density by class of statistics based on spacing," Scand. J. Statist., 1992.
14. M. Goria, N. Leonenko, V. Mergel, and P. L. N. Inverardi, "A new class of randomvector entropy estimators and its applications in testing statistical hypotheses," Nonparametric Statist., 2004.
15. E. Liitiäinen, A. Lendasse, and F. Corona, "On the statistical estimation of rényi entropies," presented at the presented at the IEEE/MLSP 2009 Int. Workshop Mach. Learn. Signal Process., Grenoble, France, Sep. 2-4, 2009.
16. Q.Wang, S. R. Kulkarni, and S. Verdú, "Divergence estimation of continuous distributions based on data-dependent partitions," IEEE Trans. Inf. Theory, vol. 51, no. 9, pp. 3064-3074, Sep. 2005.
17. I. Ahmad and P.-E. Lin, "A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.)," IEEE Trans. Inf. Theory, vol. IT-22, no. 3, pp. 372-375, May 1976.
18. P. B. Eggermont and V. N. LaRiccia, "Best asymptotic normality of the kernel density entropy estimator for smooth densities," IEEE Trans. Inf. Theory, vol. 45, no. 4, pp. 1321-1326, May 1999.
19. P. J. Bickel and Y. Ritov, "Estimating integrated squared density derivatives: Sharp best order of convergence estimates," Sankhya: Indian J. Statist., vol. 50, pp. 381-393, Oct. 1988.
20. P. Hall and J. S. Marron, "Estimation of integrated squared density derivatives," Statist. Prob. Lett., vol. 6, pp. 109-115, 1987.
21. L. Birge and P. Massart, "Estimation of integral functions of a density," Ann. Statist., vol. 23, no. 1, pp. 11-29, 1995.
22. E. Giné and D. Mason, "Uniform in bandwidth estimation of integral functionals of the density function," Scand. J. Statist., vol. 35, pp. 739-761, 2008.
23. M.M. V. Hulle, "Edgeworth approximation ofmultivariate differential entropy," Neural Comput., vol. 17, no. 9, pp. 1903-1910, 2005.
24. X. Nguyen, M. J. Wainwright, and M. I. Jordan, "Estimating divergence functionals and the likelihood ratio by convex risk minimization," IEEE Trans. Inf. Theory, vol. 56, no. 11, pp. 5847-5861, Nov. 2010.
25. B. Laurent, "Efficient estimation of integral functionals of a density," Ann. Statist., vol. 24, no. 2, pp. 659-681, 1996.
26. H. Singh, N. Misra, and V. Hnizdo, "Nearest neighbor estimators of entropy," Ann. Statist., 2005.
27. D. Evans, A. Jones, and W. M. Schmidt, "Asymptotic moments of nearest neighbor distance distributions," in Proc. R. Soc. A, 2008, vol. 458, pp. 2839-2849.
28. Y. Baryshnikov, M. D. Penrose, and J. Yukich, "Gaussian limits for generalized spacings," Ann. Appl. Prob., vol. 19, no. 1, pp. 158-185, 2009.
29. D. Evans, "A law of large numbers for nearest neighbor statistics," in Proc. R. Soc. A, 2008, vol. 464, pp. 3175-3192.
30. D. O. Loftsgaarden and C. P. Quesenberry, "A nonparametric estimate of a multivariate density function," Ann. Math. Statist., 1965.
31. K. Sricharan, R. Raich, and A. O. Hero, "Empirical estimation of entropy functionals with confidence," ArXiv E-Prints, Feb. 2012.
32. Y. P. Mack and M. Rosenblatt, "Multivariate-nearest neighbor density estimates," J. Multivariate Anal., vol. 9, no. 1, pp. 1-15, 1979.
33. K. Fukunaga and L. D. Hostetler, "Optimization of-nearest-neighbor density estimates," IEEE Trans. Inf. Theory, vol. IT-19, no. 3, pp. 320-326, May 1973.
34. X. S. Raymond, Elementary Introduction to the Theory of Pseudodifferential Operators. Boca Raton, FL: CRC Press, 1991.
35. D. S. Moore and J. W. Yackel, "Consistency properties of nearest neighbor density function estimators," Ann. Statist., 1977.
36. J. Blum, H. Chernoff, M. Rosenblatt, and H. Teicher, "Central limit theorems for interchangeable processes," Can. J. Math., Jun. 1957.
37. N. Leonenko, L. Prozanto, and V. Savani, "Correction:Aclass of Rényi information estimators for multidimensional densities," Ann. Statist., vol. 38, pp. 3837-3838, 2010.
38. K. Sricharan and A. Hero, "Weighted k-nn graphs for Rényi entropy estimation in high dimensions," in Proc. IEEE Statist. Signal Process. Workshop, 2011, pp. 773-776.
39. V. C. Raykar and R. Duraiswami, "Fast optimal bandwidth selection for kernel density estimation," Proc. 6th SIAM Int. Conf. Data Mining, pp. 524-528, 2006.